DEV Community: MORINAGA

How three automated writers confused our content-publish gate with one shared label

MORINAGA — Mon, 14 Sep 2026 11:06:47 +0000

scripts/polish.py picks summaries from a deterministic pool of template sentences, seeded by the md5 hash of each model slug. It makes no LLM call. The label it stamps on every row it writes is now "polish-py-template".

Until September 2026, it stamped "claude-routine-polish". So did scripts/humanize-aiappdex.mjs. So did the Claude Code Routine that generates curated model entries via API.

Three generators, one label. The curation gate in apps/ai-tools/src/lib/curation.ts couldn't tell any of them apart.

What the gate checks

The publishable check is a single set membership test:

export const PUBLISHABLE_MODEL_USED_LIST = ["claude-routine", "human-edited"];
export const PUBLISHABLE_MODEL_USED: ReadonlySet<string> = new Set(PUBLISHABLE_MODEL_USED_LIST);

if (!m.model_used || !PUBLISHABLE_MODEL_USED.has(m.model_used)) {
  return "unpublished:model_used";
}

"claude-routine-polish" is not in that set. When all three generators stamped it, the gate was blocking all of them — including the genuinely LLM-generated rows. Template content and Claude-drafted content were both unpublished, both indistinguishable, both labelled the same way.

The three generators

Each produces meaningfully different output but all arrived at the same label:

scripts/polish.py: Pure template. An md5 seed selects from a pool of summary sentences with {n} (download count) and {a} (author) placeholders filled in. Consistent structure, no variation beyond the placeholders. Makes no API call.

scripts/humanize-aiappdex.mjs: Builds sentences from HuggingFace model metadata: pipeline tag, language list, license. Also deterministic, also no LLM call. The header of scripts/relabel-aiappdex-provenance.py describes its shape as "one of its 6 openers AND one of its 8 caveats" — a fixed opener-caveat sentence structure.

The Claude Code Routine: Calls Claude Haiku 4.5 via the Anthropic API. Produces entries with genuine prose variation, specific technical context, and actual model-level detail that the template pools don't have access to.

These three are not equivalent. A curation gate designed to ensure pages are built only from the third class was working from a label that didn't encode which class any row belonged to.

What the relabeling script found

scripts/relabel-aiappdex-provenance.py ran a template-matching pass over apps/ai-tools/src/data/models.json to assign the correct label per row after the fact. It produced four classes:

claude-routine — summary didn't match any known template shape. Residual class: likely LLM-generated but not positively identified as such.
polish-py-template — verbatim match against the scripts/polish.py sentence pools (with {n}, {a}, {ln} wildcarded).
metadata-derived — matched the humanize-aiappdex.mjs opener-plus-caveat structure.
fallback-template — the ETL's own fallback sentence, written by the ETL when no generator ran.

The script's own header is explicit about one caveat: "claude-routine" is the residual class, not a positive identification. A row lands there because the script couldn't attribute it to any known template pattern. An older revision of one of the template generators whose shape wasn't encoded in the classifier would also land there. The agreement between the relabeling script's output and an earlier generation audit isn't independent corroboration — it's the same template-matching method applied to the same corpus, so reproducibility was guaranteed.

What this looked like from the output side

The publish gate gates page generation, not data ingestion. All rows were stored in models.json regardless of model_used. But only rows labeled "claude-routine" or "human-edited" became pages.

During the period when all three generators stamped "claude-routine-polish", the rows that should have been published as pages weren't. Astro's build gate (apps/ai-tools/src/lib/curation.ts is called at build time) filtered them out silently. There was no build error — the build succeeded, it just produced fewer pages than it should have.

Silent misses of this kind are harder to notice than failures. A build error stops everything. A correctly behaving gate returning the wrong answer from a wrong label produces a running site with missing pages.

The fix is obvious in hindsight

Unique label per generator, set at write time.

scripts/polish.py now stamps "polish-py-template". scripts/humanize-aiappdex.mjs stamps "metadata-derived". The Claude Routine stamps "claude-routine". The gate's publishable set matches what the generators actually write.

The template residue detection work I wrote about earlier ran the opposite direction: infer provenance from content text after the fact. That article found the same tradeoff — text-based classification at scale is expensive and fragile compared to asserting the class at write time. Labels are cheap. Classifiers built to recover missing labels are not.

The broader lesson: any field that drives automated decisions downstream should carry the correct value at the moment of write. A label that needs a one-off relabeling script to become meaningful is a label that was wrong from the start.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Three ways a GitHub ETL can silently delete valid alternatives — and how I fixed each

MORINAGA — Mon, 14 Sep 2026 11:06:44 +0000

The OSS alternatives directory I'm building pairs each SaaS product with its open-source alternatives by fetching live data from GitHub's API. Every night a refresh job runs: it upserts the repos it can reach and deletes the ones that are no longer in the current seed list.

Deleting stale data is the right default. A repo that disappears from the seed was removed for a reason — the SaaS page it belonged to is gone, or the alternative was curated out. But the delete logic I had originally couldn't distinguish between "no longer in the seed" and "temporarily unreachable." Those are different states that produce the same symptom: the repo is missing from the keep array the DELETE uses as its NOT IN set.

I found three distinct ways that deletion was silently wrong. None of them produced an error. All of them would eventually cause data loss.

Failure 1: An incomplete keep-list from one failed fetch

The refresh loop works like this: for each SaaS entry, fetch each alternative from GitHub, push the successful full_name values into a keep array, then run DELETE FROM alternatives WHERE saas_slug = ? AND lower(repo) NOT IN (keep).

If GitHub returns a 403 or 429 for one of five alternatives, that repo doesn't make it into keep. The DELETE then removes it from the database even though it's in the seed and the only reason it's missing is a transient API error.

The next run — if it fetches successfully — re-inserts the entry. So the visible symptom is flickering: an alternative is present one day, absent the next, present again. Hard to notice unless you're watching row counts.

The fix is a failed counter per SaaS slug:

if (failed > 0) {
  console.warn(
    `  ! ${s.slug}: ${failed} fetch failure(s) — skipping stale-row prune for this slug`
  );
} else if (keep.length > 0) {
  const placeholders = keep.map(() => "lower(?)").join(", ");
  await db.execute({
    sql: `DELETE FROM alternatives WHERE saas_slug = ? AND lower(repo) NOT IN (${placeholders})`,
    args: [s.slug, ...keep],
  });
}

If any fetch in the slug's loop fails, the DELETE NOT IN prune is skipped entirely for that slug. The entries stay in the database until a clean pass — one where every fetch succeeds — confirms the current ground truth.

This failure pattern appears in other forms. The 94-day detection lag in one of my pipelines came from a Reddit source that returned 403 and whose catch handler returned an empty array instead of re-throwing. An empty array is not obviously wrong. A 403 that's caught and silenced isn't obviously wrong either. The fix in both cases is identical: distinguish "successfully queried and found nothing" from "query failed, and I don't know what's there."

I run the same principle in a different domain: a shelf-scanning project running on a Raspberry Pi 3. Rather than alerting on any single scan that detects a gap, the system uses a temporal majority vote — a detection is only confirmed if it appears in at least 2 of the last 3 scans (the three post-processing layers are described here). The GitHub ETL's failed counter is the same principle: don't commit a DELETE based on a set of observations you know is incomplete.

Failure 2: Seed casing vs GitHub canonical naming

The seed file lists alternatives by GitHub repo path — Requarks/wiki, calcom/cal.com, and so on. GitHub's API returns a full_name that reflects the current canonical spelling, including the current owner and exact casing. The problem is that repos get renamed, owners change, and the seed spelling drifts away from what GitHub considers the authoritative identifier.

Original code used the seed path as the database primary key. Two bugs followed:

Duplicate rows: A repo appears as calcom/cal.com in one seed version and differently later. The next run inserts under the new spelling without removing the old row. A 2026-09 audit of the directory pages found 4 pages with duplicate alternative listings from exactly this casing drift.

Self-referencing pages: Two pages had an alternatives row pointing back to the same product as the page's SaaS entry — a repo ended up in its own product's list. These are in the audit notes as "自己参照 2 頁" (self-reference 2 pages).

The fix uses r.full_name from the API response as the database key:

const canonical = r.full_name; // resolved after GitHub rename/transfer
await db.execute({
  sql: `INSERT INTO alternatives (saas_slug, repo, ...) VALUES (?, ?, ...)
        ON CONFLICT(saas_slug, repo) DO UPDATE SET
          name = excluded.name, stars = excluded.stars, ...`,
  args: [s.slug, canonical, ...],
});
// Remove any row stored under a different casing or old name for this repo
await db.execute({
  sql: `DELETE FROM alternatives
        WHERE saas_slug = ? AND repo <> ?
          AND (lower(repo) = lower(?) OR lower(repo) = lower(?))`,
  args: [s.slug, canonical, canonical, repoFull],
});

The second DELETE fires on every successful fetch. It catches the case where the DB held the seed spelling and the API returned the canonical — removing the stale-casing row immediately after writing the canonical one. This is different from the NOT IN prune in failure 1: it's a per-repo cleanup on success, not a per-slug cleanup at the end of a loop.

The relationship to ON CONFLICT patterns is worth naming. DO UPDATE SET is correct here because the API's response (current stars, last push date) is fresher than what was in the DB. Using DO NOTHING instead would mean repos that existed under the old casing would never get their data refreshed. The right strategy depends on which version of the data is authoritative, and the API is authoritative over the seed for live metrics.

The GitHub API license fields I described earlier have a similar property: what the API says about a license is authoritative over what I infer from other signals, and the DB should reflect the API's answer, not my seed's guess.

Failure 3: Unguarded bulk delete when the seed file shrinks

A separate function, pruneStaleSaas, handles SaaS-level cleanup: if a product is removed from the seed, its rows (SaaS record, content, all alternatives) should be removed from the database.

The logic is straightforward: read all SaaS slugs from the DB, filter out those present in the current seed, delete the rest. This works when the seed is intact. When the seed file is accidentally truncated — a merge conflict that ate half the JSON, an editing mistake — the "stale" list becomes most of the database.

The fix is a ratio guard:

const MAX_STALE_SAAS_RATIO = 0.1;
const MAX_STALE_SAAS_FLOOR = 3;

const limit = Math.max(
  MAX_STALE_SAAS_FLOOR,
  Math.floor(rows.rows.length * MAX_STALE_SAAS_RATIO)
);
if (stale.length > limit) {
  console.error(
    `  ! ${stale.length} saas rows not in seed exceeds prune limit ${limit} — skipping prune (check seed-saas.json)`
  );
  return;
}

If more than 10% of DB rows appear stale — or more than 3 rows when the database is small — the prune is skipped and an error is logged. Between runs, the seed almost never loses more than 10% of entries legitimately. A stale count that high is almost certainly a data anomaly.

The floor of 3 handles early-stage databases: at 5 rows, 10% would be 0, which would fire the guard on a single legitimate removal. The floor ensures small databases can still prune.

This is a circuit-breaker pattern, not a repair. It stops the damage and logs something visible. The three approaches to silent failure detection I've written about all share this structure: detect the anomaly, stop, make noise. The code doesn't try to infer whether the seed file is valid — it just checks whether the prune looks plausible given current DB state.

The shared principle

Looking at the three fixes together: before any DELETE, verify that the input driving the DELETE is trustworthy.

Failure 1: the keep list must be complete (no failed fetches). If it's not, skip the delete.
Failure 2: the repo identity must be resolved through the API's canonical name, not the seed's spelling. Use the API answer as the key.
Failure 3: the "absent from seed" count must be plausible given current DB size. If it's not, skip the bulk prune.

None of these changed the delete logic itself. They're preconditions: checks on whether the delete instruction is based on trustworthy inputs before it runs.

A practical rule from this: any DELETE that uses an externally derived set (API results, a seed file, a user-supplied list) should have an explicit check on whether that set is complete before running. If completeness can't be verified, defer to the next run. A stale row persisting an extra night is much cheaper than a valid row vanishing without an error message.

FAQ

Why not just log errors instead of skipping deletes?

The errors are logged — console.warn names the slug and the failed count. But logging doesn't protect the rows. The guard and the logging are both needed: the guard prevents data loss, the log makes the gap visible.

What if the failed counter fires on a genuinely deleted repo?

If GitHub returns 404 for a repo that's actually gone (not temporarily unavailable), the failed counter fires and the prune is skipped, leaving the deleted repo in the database for one more night. The tradeoff is deliberate: a falsely persistent row is recoverable; a falsely deleted row isn't (without log archaeology). The next time the seed is updated to remove that repo, the per-slug cleanup will handle it.

How does the ratio guard handle a legitimate large cleanup?

If a batch seed cleanup removes more than 10% of entries at once, the guard blocks the prune. The fix is to either raise the ratio temporarily or run the prune manually. The guard is a circuit breaker on unexpected deletions, not a permanent cap on how many entries can be legitimately removed.

Does canonical full_name resolution require an extra API call?

No. The full_name is in the same response as stars, description, and license — there's no additional call per repo. The 100ms sleep between calls was already present for rate-limit reasons, not added for the canonical lookup. For reference, GitHub's authenticated REST API rate limit is 5,000 requests per hour, so 100ms (36,000 per hour at max throughput) is well inside the safe range.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Four production models I'm watching: speaker diarization, reranking, OCR, prompt safety

MORINAGA — Sun, 13 Sep 2026 10:45:25 +0000

I pull model stats weekly to keep aiappdex.com current. Most entries drift in with download counts that tell a straightforward story: high downloads, modest likes, task-category role obvious. Four recent models have a different profile — either their likes-to-download ratio is unusually high, or their pipeline tag puts them in infrastructure territory that rarely gets covered in round-ups.

None of these are chat models. All four appear in production systems at scale.

pyannote/speaker-diarization-3.1 — 9.26 million downloads, 3,350 likes

speaker-diarization-3.1 on aiappdex answers a question that Whisper does not: who spoke when. Transcription tells you what was said; diarization adds speaker segmentation and identity labels across the recording.

The likes-to-download ratio here is exceptional — one like for every 2,763 downloads. For context, most embedding models in the same download range hover around one like per 20,000 or more. That high ratio usually means specialists are using the model and are grateful enough to explicitly endorse it. The use cases are narrow but high-value: meeting transcription with attribution, podcast chapter editing, call centre analytics where GDPR and speaker count matter. Those are contexts where an open-weight option with Apache 2.0 licensing has obvious appeal over a cloud API.

The pipeline is a two-stage system — segmentation, then speaker embedding and clustering. Pyannote's own documentation covers the necessary speaker count parameter; most production deployments I've seen in open-source tooling treat that parameter as a config value rather than hardcoding it.

BAAI/bge-reranker-v2-m3 — 17.58 million downloads, 1,161 likes

bge-reranker-v2-m3 on aiappdex is not the same model as BGE-M3, though the names overlap. BGE-M3 is a bi-encoder embedder; this is a cross-encoder reranker. The difference matters at runtime: a bi-encoder encodes documents once and caches the results; a cross-encoder re-encodes every query-document pair at query time, which is slower but more accurate.

The standard RAG pattern that reaches for this model: a fast bi-encoder retrieves the top 100 candidates, then bge-reranker-v2-m3 re-scores the top 20 for the final ranked list. What distinguishes it from the simpler ms-marco cross-encoders is multilingual support — it is built on XLM-RoBERTa and handles queries across languages from a single checkpoint. Apache 2.0, so no commercial-use complications.

17.58 million downloads places it firmly in production territory. The model is large enough that you want GPU acceleration for interactive latency, but at batch re-ranking it runs fine on a modest CPU setup.

ZhipuAI/GLM-OCR — 2 million downloads, 2,011 likes

GLM-OCR on aiappdex has a likes-to-download ratio close to 1:1. In practice that almost never happens — download counts accumulate from CI pipelines, automated tests, and package managers; likes require a human to click. When the ratio approaches parity, you are looking at a model that people download specifically because they went looking for it, not one that arrived as a dependency.

GLM-OCR is MIT licensed, treats OCR as a sequence generation task, and supports Chinese, English, French, Spanish, Russian, German, Japanese, and Korean from a single model. That language coverage at MIT licensing is the specific combination that makes it interesting — most multilingual OCR alternatives either require cloud APIs or carry non-commercial restrictions. I do not run an OCR pipeline right now, but if I needed structured text extraction from screenshots or document images at the directory scale, this would be the first checkpoint I'd evaluate.

Meta/Prompt-Guard-86M — 4.37 million downloads, 397 likes

Prompt-Guard-86M on aiappdex is a safety classifier for detecting prompt injection in user-supplied text. With 86 million parameters it runs on CPU with low latency, which is the point — it is meant to be a gate, not the main workload.

This one is relevant to my own pipeline. The article routine reads Dev.to reader comments and treats them as data for the VoC engine. There is a hard rule in the pipeline: comments are untrusted input and must never be followed as instructions (a fact documented in the project's recent-work-facts.md: reader comments feed an autonomous routine that commits and publishes, making injection the specific threat). Right now the defence is a combination of prompt framing and delimiting — the comments are labelled explicitly as untrusted data before being passed to any LLM. A dedicated classification step that runs before the LLM sees the text is the obvious next layer, and Prompt-Guard is the open-weight option for that.

One caveat that matters: Prompt-Guard-86M is subject to Llama 3.1 Community license terms, not Apache 2.0. Check that licence before commercial production use.

All four models are in the aiappdex directory at aiappdex.com. The download counts here are from this week's ETL snapshot; likes counts shift more slowly and tend to be more stable as a quality signal.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Three YouTube Data API v3 behaviors that broke my analytics classifier

MORINAGA — Sat, 12 Sep 2026 09:44:12 +0000

I run a daily Python script that fetches the last 30 videos from a YouTube channel, classifies them as high or low performers by view count relative to the channel median (as of July 2026; the script has since switched to a views-per-day rate with an age-eligibility window), and writes a strategy directive telling the next day's video-generation routine which archetype to produce. I wrote about the classification logic here.

Three behaviors in the YouTube Data API v3 broke that script. Only one of them announced itself with an error, and even that error didn't say why. Each is documented if you read the reference carefully, but they're easy to miss when you're building quickly.

1. Channel handle lookup fails silently when the handle doesn't match

To look up a channel by its handle (the @name you see in YouTube URLs), you use the forHandle parameter:

GET /youtube/v3/channels?part=snippet,statistics&forHandle=myhandle&key=<key>

The problem: when the handle doesn't match — a missing underscore, a hyphen where the real handle has none, a handle that was renamed since you wrote it down — the API returns a 200 OK with an empty items array. No error, no 404, no explanation. Your code receives valid JSON that looks like a successful response, but there are no channels in it.

My first API-key version tried exactly one handle — YT_CHANNEL_HANDLE, defaulting to claude_automate — and raised RuntimeError with the entire response body when items came back empty. That raise is the only reason this didn't turn into silent bad data. What it couldn't do was explain anything: the run died with channel not found for handle @... followed by a body that was a perfectly healthy 200 OK carrying an empty array. The error proved the request had succeeded and told me nothing about which string form the channel actually answers to, and I never did pin down the exact mismatch — I stopped trying to identify it and made the lookup stop depending on getting it right.

The fix is a cascade: try multiple handle variants (claudeautomate, claude_automate, claude-automate alongside the configured one), then fall back to a search.list query if all handle lookups return empty:

for handle in [env_handle, "channelname", "channel-name", "channel_name"]:
    result = api_get(f"channels?part=snippet&forHandle={handle}&key={key}")
    if result.get("items"):
        return result["items"][0]

# Final fallback: search by channel name
search = api_get(f"search?part=snippet&type=channel&q=channel+name&key={key}")
for item in search.get("items", []):
    channel_id = item["snippet"]["channelId"]
    result = api_get(f"channels?part=contentDetails,statistics&id={channel_id}&key={key}")
    if result.get("items"):
        return result["items"][0]

The search fallback is slower (one extra API call) and counts against quota, but it's a last resort for a once-daily job. The real lesson is that any YouTube handle lookup needs a fallback path — there's no single canonical string form that reliably works across all channels.

The forUsername parameter is also worth knowing: it's the legacy form for older channels that predate the @handle system. If your channel predates 2022 or so and forHandle keeps failing, try forUsername with the channel's original username.

2. View counts require a second API call

playlistItems.list gives you the list of videos in a channel's uploads playlist. It includes video IDs, publish timestamps, titles, and descriptions. It does not include view counts, like counts, or any performance statistics.

To get viewCount, likeCount, and commentCount, you need a separate videos.list call with part=statistics:

def fetch_uploads(api_key, uploads_playlist_id):
    # Returns: videoId, publishedAt, title, description
    # Does NOT return: viewCount, likeCount, commentCount
    url = f"playlistItems?part=snippet,contentDetails&playlistId={uploads_playlist_id}&maxResults=30&key={api_key}"
    items = api_get(url).get("items", [])
    return [item["contentDetails"]["videoId"] for item in items]

def fetch_stats(api_key, video_ids):
    ids = ",".join(video_ids)  # batch up to 50 IDs per request
    url = f"videos?part=statistics,snippet&id={ids}&key={api_key}"
    return api_get(url).get("items", [])

The videos.list endpoint lets you batch up to 50 video IDs per request as a comma-separated string. For a 30-video sample, one call gets everything.

One thing to watch: the order of items in the videos.list response is not guaranteed to match the order of IDs in the request. Don't assume stats[i] corresponds to video_ids[i]. Match by item["id"]:

stats_by_id = {v["id"]: v for v in fetch_stats(api_key, video_ids)}
ordered = [stats_by_id[vid] for vid in video_ids if vid in stats_by_id]

The two-call minimum also means you need the contentDetails.relatedPlaylists.uploads value from the channel response (the uploads playlist ID), which is its own separate fact. Getting from "channel handle" to "view counts on the last 30 videos" requires: channels.list → playlistItems.list → videos.list. Three calls minimum. The YouTube Data API v3 videos.list reference documents which part values return which fields — statistics is the one that has view counts.

3. Including "unknown" videos corrupts the archetype ranking

After getting view counts, I join each video with its archetype — product_findindiegame, build_in_public, meta, and so on — by matching the YouTube title against titles stored in local uploaded-video JSON files. The match uses word overlap: if at least 4 content words are shared between the API title and the local title, the video gets that archetype. If no match clears the threshold, the video gets _archetype = "unknown".

The initial version kept "unknown" in the pool and picked the next day's preferred archetype by frequency — the most common archetype among the high performers. Here's the archetype table from the 2026-06-20 report, the one that made the problem obvious:

archetype        |  n | median_views
-----------------|----|-------------
unknown          | 22 | 19
contrarian       |  2 | 15
curated          |  1 | 10
build_in_public  |  1 | 55
meta             |  1 | 18
ai_tools         |  1 | 7
technical        |  1 | 67
recap            |  1 | 1

unknown wasn't the best-performing bucket — technical (67) and build_in_public (55) both beat its median of 19. It was simply the biggest bucket: 22 of the 30 sampled videos, and 9 of the 11 high performers. So the frequency-based tuner emitted "Prefer: unknown." It also emitted "Avoid: unknown," because unknown dominated the low bucket too. Contradictory and meaningless guidance — "unknown" isn't a content format you can produce, and while the tuner was chasing it, genuinely weak formats (a recap video sitting at 1 view) kept getting generated.

The reason "unknown" swallowed the sample: it collected the older videos that predated the automated pipeline (no uploaded JSON file exists for them) and recent videos whose archetype file never got written, because the publish workflow had been failing on artifact quota. Neither group says anything about which format to produce tomorrow.

The fix was two changes in one commit: rank archetypes by median views instead of by how often they show up in the high bucket, and drop "unknown" from the ranking entirely:

for video in videos:
    archetype = video.get("_archetype", "unknown")
    if archetype == "unknown":
        continue  # skip — no actionable signal
    arch_views[archetype].append(int(video["statistics"]["viewCount"]))

ranked = sorted(arch_views.items(), key=lambda kv: statistics.median(kv[1]), reverse=True)

After exclusion, the rankings only reflect videos where I know the production context. "Prefer product_findindiegame" is actionable. "Prefer unknown" is noise.

The same pattern comes up in any classifier built on fuzzy join data: always define your null category explicitly, and decide upfront whether it should participate in ranking or be excluded. Null-inclusive rankings silently inflate or deflate whichever real category happens to share data with the null bucket.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Five things I noticed this week: deletion bias, hidden assumptions, production AI

MORINAGA — Sat, 12 Sep 2026 09:44:08 +0000

Five observations from this week. Loose format — things that surprised me, contradicted something I assumed, or changed how I think about a problem.

1. I deleted my way to a fake improvement in YouTube analytics

I delete YouTube shorts that fall below 50 views in their first week. The reasoning: remove clutter, sharpen the signal. What I hadn't accounted for: deleting the worst performers removes them from the denominator.

On 2026-08-11, the headline median jumped from 2.9 to 3.3 views/day. I started writing a note about the improvement. Then I recomputed with deleted videos carried at their last observed value (LOCF correction). The corrected figure was 2.91 — flat. The apparent gain was entirely the deletion.

It got worse with A/B testing. One arm had 36% attrition, the other 44%. The effect size swung between 0.99× and 1.89× depending only on how I handled the missing data. The test is now SUSPENDED rather than reporting a winner. Full writeup here.

The pattern has a name: outcome-dependent censoring. The measurement outcome is exactly the selection criterion for removal. Once you see the shape of the error, you see it everywhere — any system that deletes, archives, or filters its worst observations before aggregating them will drift upward for free.

2. Cadence assumptions are the sneakiest hidden state in GitHub Actions

I published a piece on systematic review of GitHub Actions workflows — running two independent review passes to surface assumptions an author wouldn't notice. Two passes across a batch of workflows found 8 defect categories.

The sneakiest: assumptions tied to cadence. A cleanup window set to 7 days implicitly assumes the generating job also runs daily. A retry backoff of "a few minutes" implicitly assumes the trigger fires less than once per hour. Neither shows up as an explicit constraint in the code. They're encoded in the numbers.

A reader on that article: "every assumption about external systems eventually decays when the load pattern changes." That matches what I've seen. The code is unchanged; the inputs changed. The defect was already there — it just needed the right timing to become visible.

3. The headless Pi scanner hit the same class of implicit constraint

On the edge AI shelf scanner I run on a Raspberry Pi 3, one issue I had to handle explicitly after an audit pass was scan collision — a new cron run starting before the previous scan finished. At median 8.5 seconds per inference on an hourly cycle, it almost never matters. But SD card writes, post-processing, and occasional outliers (range went to 11.8 s across 19 real scans) can push the total runtime higher.

The collision case wasn't in any spec. It was implicit in the assumption that "hourly scans are fine" — which is true until they aren't, and the failure mode isn't a crash, it's corrupted scan history. Recognizing it required thinking through what overlapping runs would actually do, not just whether the timing worked on average.

Same shape as the GitHub Actions cadence problem: the constraint was real but invisible until someone asked "what breaks at the boundary?"

4. Production HuggingFace downloads are dominated by models nobody blogs about

Two pieces this week on what actually leads production downloads in the aiappdex model catalog:

all-MiniLM-L6-v2 has 246 million downloads. BERT-base-uncased from 2018 has 58.6 million. Both sit above any text-generation model in the catalog. The non-text production models — CLIP, MobileNetV3, Chronos-T5-Large for time series — have their own substantial download bases and cover use cases that LLMs don't touch.

The models that dominate AI news — frontier chat models, large instruction-tuned LLMs — don't appear near the top of production download counts. This isn't a criticism of coverage. It's a gap between what gets discussed and what gets used: embedding models and smaller classification baselines still power most production inference. The discourse is shaped by what's interesting to write about.

5. Detection lag tells you more than uptime ever did

This week's piece on detection lag vs. uptime covers something I keep coming back to: six failures in this repo with detection lags ranging from 36 to 113 days, all while the jobs stayed green.

The common pattern: the job produced a well-formed output file. An empty array is not an error. A file with wrong numbers isn't a parse failure. A published article that nobody reads doesn't throw an exception. The job finished, the badge went green, nothing in the monitoring flagged it.

Measuring how long it takes to notice a failure — not whether the job ran — gives a more honest picture. It also tells you where to invest monitoring effort: not at the process boundary (did the job exit 0?) but at the output boundary (does the content of this file look plausible?). The pipeline health question isn't "is it running?" It's "when it breaks, how long until I know?"

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Three Signals from 284 Published Articles That Changed How I Plan New Content

MORINAGA — Fri, 11 Sep 2026 10:08:15 +0000

When I started cross-publishing articles in April 2026, the dataset was too small to tell me anything useful. I had 10 articles and single-digit reactions — not a sample, just noise. After 284 articles — machine-counted by scripts/article-analytics.mjs querying the Dev.to public API daily — there are patterns worth naming. Three of them changed what I actually do when planning new posts.

One caveat upfront: 284 articles with 173 total reactions is a heavily right-skewed distribution. Most articles have zero engagement. The patterns here describe the exceptions, and the exception count is still small. These are directional signals, not controlled results.

Signal 1: Recent articles outperform older ones, despite less time to accumulate

Articles published in the last 30 days average 1.26 reactions each. Articles older than 30 days average 0.42 reactions. The older posts have had months longer to accumulate, so this runs opposite to what you'd expect from a compounding model where traffic builds over time.

The most plausible explanation: Dev.to's feed surfaces recent content, and reaction behavior follows that surfacing. An article from three months ago sitting in search results doesn't receive the same passive impressions that a new post gets from the "latest" and "following" feeds. The engagement window is probably the first several days after publication, not the full lifetime of the post.

What this changed: I stopped treating the backlog as a long-term asset that compounds quietly. Articles are more like timestamped snapshots — the publication moment is when they reach most of the audience they'll ever reach. Publishing consistently matters more than perfecting any individual article before shipping it.

Signal 2: Weekly-digest format has the highest median reactions

Across all format archetypes in the 284-article set, weekly-digest posts (the "five things I noticed this week" format I run on Saturdays) have the highest median reaction count: 1.5. Technical howto posts and "why I'm betting on X" meta articles occupy the top 10 individually, but the digest posts do it more consistently across the format as a whole.

My interpretation: digest posts have lower reading commitment than a 2,000-word technical deep dive. Someone who wouldn't read a full article on canonical URL implementation might still spend 90 seconds on a digest that mentions it alongside four other observations. The format matches available attention better than a single long argument does.

What this changed: I stopped treating digest posts as lower-value output. They're the most consistently engagement-optimized format I have. The trade-off is that they have less long-tail keyword potential than a specific technical article — a digest titled "Five things I noticed this week" won't rank for anything in six months. Digest posts serve current readers; technical articles serve future search. Both have a role, and I was underweighting the digest.

Signal 3: Concentration is extreme — two articles account for a disproportionate share

The top article has 5 reactions and 5 comments. The second has 4 reactions and 4 comments. Together that's 13 of the 173 total reactions — 7.5% of engagement concentrated in two of 284 articles.

More interesting than the concentration is what the top two articles are about. "Four signals I built into an OSS decision score instead of fabricating reviews" and "How I coordinate Claude and Codex sessions with a pull-based JSON handoff ledger." Neither is about business strategy or AI tools in general. Both are about specific engineering decisions with concrete alternatives implied.

The OSS scoring article has a number in the title — four signals — and describes something that can't easily be found elsewhere (an actual scoring approach, with the decision to avoid fabricating reviews stated explicitly). The coordination article names a mechanism — pull-based JSON — that immediately suggests its alternative (push-based, or a database). Both titles contain something falsifiable and specific.

What this changed: I now try to make titles contain at least one concrete number or mechanism. Not as a formula — the dataset is too small for formulas — but as a heuristic: the articles that got the most engagement in my own data all have a specific, verifiable claim in the title. That's directional evidence, held loosely.

What I'm not changing

Despite the recency signal, I'm not shifting to more-frequent shorter posts at the expense of technical depth. The engagement feedback loop I built is meant to surface which formats are connecting with readers, not to optimize for Dev.to feed placement at the cost of the kind of writing I'd still want to have published in month 12.

The engagement data is useful for calibration, not direction. A recency effect in the feed is real but shallow — it doesn't tell me that shallow content outperforms deep content, it tells me that content visible in the feed outperforms content not visible in the feed. The goal is to be consistently visible, which means publishing regularly, which I was already doing.

The one concrete change: running this engagement report daily and checking it weekly. The data comes from the Dev.to (Forem) public API — specifically the articles endpoint which returns reaction and comment counts without authentication for published posts. Before I built the automated collection, I had no systematic view of what was connecting. Now I do, and the three signals above came directly from having the data in front of me in a format I could actually read.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

What I Learned About Declaring Source-of-Truth Per Claim in an Automated Pipeline

MORINAGA — Fri, 11 Sep 2026 10:08:12 +0000

After publishing Two times an external API overruled my own repository, a reader left a question I couldn't stop thinking about. @raknaos asked whether I had encoded the authoritative-source mapping anywhere, or if it was still implicit in each checker — and noted that declaring it per claim starts to look like a small trust database with its own maintenance cost.

Short answer: it's still implicit. Longer answer: the failure modes that motivated the original article aren't isolated. I found two more cases in the same pipeline with the same shape. The per-claim approach is what I should have built; this article is about why, and what it would look like.

The thesis in one sentence: when multiple data sources can each describe the same claim, and an automated checker must pick one, the choosing must be declared in configuration — not inherited as a convention from which source happens to be easiest to query.

The class of failure: confident wrong corrections

Both failures from the previous article had this structure: an automated checker read a claim, found evidence from its primary source, and updated the claim to match. The update looked correct from the checker's perspective. From the article's perspective, it introduced an error.

This isn't a quality-gate problem in the usual sense. The quality contract v2 frontmatter I added to every article includes verified_at — a date for when claims were checked. That field tells me a check ran; it doesn't tell me which source was authoritative. Two auto-correction passes using different sources can each attach a valid verified_at to contradicting facts, and both timestamps look legitimate.

The problem is not "did the checker run?" It's "which source did the checker consult, and was that the right one for this particular claim?" A green verified_at doesn't answer that.

Two cases from my pipeline

Case 1: Rejection count. A published article described the project's rejection history. The repo had four events committed. The author's own notes showed seven — three rejections arrived as verbal notifications and were never committed. A checker reading git saw four as the ground truth and updated any article showing seven. A different pass reading another source saw seven and corrected it back. Each pass was confident. Each correction was wrong half the time.

The claim "how many rejections did this project receive?" has one correct answer. The problem: two sources (git commits; author notes) gave different answers, and each looked authoritative to whichever checker happened to query it.

Case 2: Newsletter status. An article described an active manual newsletter workflow. Git commit history showed publishing setup work. The Beehiiv API, when queried, showed zero posts, zero drafts — the publication had been configured but never used. The claim was settled by the source the automatic checker hadn't been reading.

The claim "is the newsletter active?" has one correct answer. The problem: git commits documented intent (the setup work), while the provider API documented outcome (zero sends). These are subtly different propositions, but the article stated the outcome, so only the outcome source was authoritative.

In both cases the failure signal was invisible. The job produced a well-formed file. The updated number looked plausible. The output inspection principle says to verify the output rather than the job status — but even inspecting the output file doesn't catch "this number looks right but came from the wrong source."

What per-claim source declaration looks like

This is the design I haven't built yet.

The minimum viable form: a configuration that lists facts with multiple plausible sources and names the authoritative one for each:

rejection_count:
  authoritative: author_notes
  fallback: none
  reason: git records committed events only; verbal notifications stay in notes

newsletter_active:
  authoritative: provider_api
  fallback: none
  reason: git records intent; the API records outcome

The checker reads this config before querying any source. It queries only the declared authoritative source. If that source is unavailable, it returns unknown — not a silent fallback to whatever is readable. The failure-closed behavior is load-bearing: a silent fallback to git is exactly what created the four-vs-seven pattern in Case 1.

The config lives in the repo. Changes to it are in git history, reviewable, attributable — the same auditability that systematic review of implicit assumptions provides for GitHub Actions configuration.

A parallel from a different project makes the value clear. The shelf scanner I run on a Raspberry Pi 3 uses NCNN for inference — a lightweight framework designed for embedded hardware. The YOLO model detects empty space — single class, no product identification. Which SKU belongs in that gap is resolved by a planogram (a shelf-layout master file), not the model. Two distinct sources, two explicit roles: the model is authoritative for "is there empty space here?"; the planogram is authoritative for "what should be stocked there?" Conflating them would require the model to do SKU recognition — a much harder problem that needs per-store retraining. The explicit boundary makes the system maintainable in a way a conflated design wouldn't. I want the same clarity in the article pipeline.

The maintenance overhead question

@raknaos framed this accurately: a per-claim trust config is a small database with its own maintenance cost. Add a new data source and you need to update the config. Change which source is authoritative for a claim and you need to reason through every place the old source was cited. An unmaintained config that points to a deprecated field is worse than no config — it routes confidently to the wrong place.

Against that: what's the cost of the current approach?

The two-corrections case produced no visible error signal. Each pass produced a well-formed file with a plausible-looking number. I found the discrepancy manually, while fact-checking something else. The Dev.to API sync behaviors I've documented are similar: some fields have exactly one plausible source and are safe to treat as authoritative implicitly; others — like reaction counts on a published article — have a correct value that only the live API knows, not a cached projection. The difference between those two categories is currently in my head, not in the code.

My rough estimate: the maintenance overhead of keeping a trust config current is lower than the debugging overhead of a confident-wrong correction in a pipeline that runs unattended. But the answer depends on how often authority shifts. In a stable pipeline reading from a fixed set of sources, the config is set once and rarely touched. In a pipeline with active ETL from multiple third-party APIs — where a provider can deprecate a field, add a more-granular endpoint, or change what a field means — the maintenance cost rises, and the config earns its overhead most visibly precisely when authority shifts.

Where implicit mapping stays acceptable

Not every claim needs a declaration. Some have exactly one plausible source:

Article word count: the file is the source, and no external authority can override it.
Publication timestamp: the platform API records it; the repo stores it after the fact. One source.
Inference time measured on your own device: your measurements, no alternative.

These don't need declarations because "which source?" has a single obvious answer. Implicit is fine when there's nothing to declare.

The pattern that warrants explicit declaration: whenever a developer could construct two plausible stories about the same fact from two different sources, and both stories would sound defensible, and an automated system must pick one without asking. That is the decision point the config exists to make explicit and auditable.

Looking back at what the quality contract v2 implementation actually introduced: verified_at records when verification ran, and original_evidence records what was checked. Neither records which source was authoritative when multiple sources exist. Adding a per-claim source declaration field to the contract is the next logical step.

What I would do differently

The verified_at field was the right idea applied at the wrong granularity. It stamps the article with "verified on this date" but doesn't record what source was queried for each claim. Two articles verified the same day from different sources can contradict each other; both have valid timestamps.

What I'm going to build: a source_declarations section in the frontmatter contract for claims that have multiple plausible sources. Each entry names the claim, the authoritative source, and a short reason. The checker script enforces it: if a claim type has an entry in source_declarations, the checker queries only that source and fails closed if it's unavailable.

This won't prevent every wrong correction. It will prevent the specific class of error where an automated checker picks the accessible source over the authoritative one, and does so silently, and the output looks correct.

FAQ

What if the declared authoritative source is unavailable when the checker runs?

Return unknown and skip the update. A checker that silently falls back to a secondary source reintroduces the failure mode you're designing around. An article with one field missing is recoverable; an article confidently carrying a number from the wrong source can sit there for weeks before anyone notices — or it can trigger a correction pass that confidently restores the wrong number again.

How do you identify which claims need declarations before you've had a failure?

You mostly don't, which is why the retroactive approach is more practical than prospective enumeration. Every confident-wrong correction reveals a claim that needed a declaration. Keep a running list. Each failure is free documentation of a gap in the config.

Is this the same as data lineage tracking?

Related but narrower. Lineage tracks how a value was transformed: "this field was computed from X using function Y at time Z." Per-claim source declaration answers a simpler, earlier question: when I need to verify this specific claim, which source do I query? The two can coexist and reference each other; neither replaces the other.

What did you actually implement as of this writing?

Nothing beyond verified_at and original_evidence in the frontmatter. The per-claim config is the next step. Writing this post is how I'm committing to the shape of it before building it. The two-corrections failure is documented in the prior article; this is the proposed design for making that failure structurally impossible next time.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Chronos-2, CLIP, and MobileNetV3: three production AI models for non-text workloads

MORINAGA — Thu, 10 Sep 2026 10:11:04 +0000

When I sort the model catalog at aiappdex.com by total downloads, the 15th most downloaded model processes time-series data. The 14th handles zero-shot image classification. The 16th runs on hardware with under 3M parameters. None of them generate text.

Three models from that tier deserve attention — not because they're new, but because the download numbers say someone is shipping them in production:

Model	Downloads	Task
Chronos-2	25,699,844	time-series-forecasting
CLIP-vit-base-patch32	19,936,700	zero-shot-image-classification
MobileNetV3-small	17,428,712	image-classification

Numbers from apps/ai-tools/src/data/models.json as of this writing.

Chronos-2 — zero-shot time-series at 25.7M downloads

Chronos-2 is Amazon's second-generation pretrained model for zero-shot time-series forecasting. The architecture frames the problem as language modeling: quantize the time-series values into tokens, run a T5 encoder-decoder, decode the output tokens back into predicted values. Apache 2.0.

The zero-shot claim is the practically important part. Most forecasting approaches require labeled in-domain training data — you need historical examples for the domain you're forecasting. Chronos-2 was trained on a large corpus of diverse time-series and can generalize to new domains without fine-tuning. That's the same premise CLIP made for vision: collect enough diverse data at pretraining time and zero-shot transfer becomes viable.

25.7M downloads. In practice this means anomaly detection and demand forecasting in production data pipelines where someone had a time series and wanted a reasonable forecast without standing up a labeling operation. That use case — "I have the data, I don't have labeled examples" — is common enough to drive those numbers.

What it doesn't do: outperform a well-tuned in-domain model on a domain where you have plenty of training data. Zero-shot is the alternative when you don't have that data, not a replacement when you do.

CLIP-vit-base-patch32 — 19.9M downloads for cross-modal search

CLIP-vit-base-patch32 is OpenAI's CLIP model with a ViT-B/32 image encoder, trained contrastively on 400 million image-text pairs. The training procedure aligns image and text representations in a shared embedding space — which means you can ask "which of these images is most similar to the phrase 'empty shelf'" without training a classifier for that specific concept.

19.9M downloads. 1,137 likes.

The B/32 variant (as opposed to ViT-L/14) trades accuracy for speed. In production that often makes sense: faster inference across more items is more useful than marginally better accuracy on fewer. The applications behind these numbers are image search, content moderation pre-filtering, and embedding images alongside text in retrieval-augmented generation pipelines.

The cross-modal property is worth highlighting separately: CLIP's image encoder produces embeddings that are directly comparable to text embeddings from the same model. That's what makes it show up in hybrid retrieval systems — you can retrieve by text query and surface matching images without a translation step.

MobileNetV3-small — 17.4M downloads for edge classification

MobileNetV3-small-100.lamb-in1k is a sub-3-million-parameter image classifier trained on ImageNet-1k via the timm library. At this size it runs on mobile hardware and embedded systems. 17.4M downloads.

MobileNetV3 is in this list for a different reason than Chronos-2 or CLIP. It isn't zero-shot and it doesn't bridge modalities. It's a very small, very fast image classifier for hardware where nothing else fits. The downloads come from the overlap of two things: timm's consistent preprocessing API (so it's easy to drop in as a swap for any other timm classifier) and the large number of production systems that need a classification signal where a 10M-parameter model is already too large.

The trade-off is explicit: you give up accuracy in exchange for a model that runs on edge hardware. For a use case where inference happens on device and latency matters more than peak accuracy, that's the right trade-off. The 103 likes (low relative to downloads) reflects that this is a utility model — people use it because it solves a specific constraint, not because they're enthusiastic about it.

The pattern across all three

None of these are frontier models. None are mentioned in the AI news I read. All three are in the top 20 downloads on a catalog that includes hundreds of text-generation and instruction-tuned models.

They're production workhorses in categories with genuinely different constraints than language generation. Chronos-2 targets the no-labeled-data case in time-series. CLIP targets cross-modal retrieval where text and images need to live in the same embedding space. MobileNetV3 targets inference on hardware where memory and latency are the primary constraints.

If you're building something in one of those three spaces, 17–26 million monthly downloads is a better production-readiness signal than a benchmark table. It means the preprocessing API is stable, the common failure modes are documented, and someone has already debugged the deployment on your target hardware.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

How systematic review catches the GitHub Actions assumptions authors can't see

MORINAGA — Thu, 10 Sep 2026 10:11:00 +0000

The conclusion I reached this week: when I changed my YouTube video pipeline from a daily cadence to three times per week, a Codex review found four defects. A second review, run the next day after I fixed those four, found four more. All eight fell into the same category. I had built assumptions about external system behavior directly into the code, and I couldn't see them because I built the system knowing how it was supposed to work. A reviewer who doesn't share that knowledge can ask questions the author doesn't know to ask.

This article is the four categories, one example each, and what I'd do differently.

The pipeline and why the cadence changed

The pipeline picks one video spec from a queue, runs a quality gate on it, and if it passes, uploads the video and commits the result back to the repository. It was running daily. After observing that the pipeline was spending publish slots on rejected specs, I cut the cadence to three times per week — Sunday, Tuesday, Thursday — to give the queue more time to fill with passing specs between runs. That's a structural change: the scheduling logic, monitoring thresholds, and gate behavior were all designed for a daily run.

I ran a Codex review on the pull request. It found four issues. I fixed those and ran a second review. It found four more. Eight total. Every single one was an assumption about how some external system behaves.

Category 1 — Monitoring thresholds tuned to the wrong cadence

The pipeline health monitor had two thresholds:

Alert if no publish in 36 hours
Open a branch-drift issue if no spec file in 2 days

Both made sense for a daily schedule. With a three-times-per-week cadence, the longest healthy gap is Friday to Monday: 72 hours. Both thresholds would have fired every Sunday on a healthy pipeline.

I hadn't noticed because I set those thresholds when the pipeline was daily. They were correct then. They felt like constants, not assumptions.

Codex caught it because it could compare the monitoring thresholds to the publish schedule without knowing "these thresholds used to be correct." It asked, in effect: what's the longest healthy gap in the new schedule? The answer — 72 hours — was longer than both alert windows. Fix: 84 hours and 4 days, giving one schedule-queueing slot of slack on each (from commit d97bc91).

Category 2 — A rejected spec consuming a publish slot

The quality gate ran on one file selected from the queue. If that file failed the gate, the job quarantined the file and exited. The slot was consumed. The next spec would try in two or three days.

At the time of the cadence cut, the queue held five specs. Two failed the gate. The pre-gate-then-pick sequence would have spent the September 11 and September 14 slots on quarantine commits and pushed the next actual upload to September 16 — a seven-day gap on a three-times-per-week schedule. Those numbers are from the commit message; I can count the files in the queue.

Codex caught this by reasoning about the queue state, not just the per-file gate logic. The per-file logic was correct in isolation. The problem was the interaction between the gate's exit behavior and the queue's current contents — a state that I knew about but that the code had no way to account for when I wrote the gate.

Fix: a pre-gate sweep step now runs first and quarantines every reject in a single commit before the pick step runs. The per-file gate stays as the fail-closed backstop.

Category 3 — Git state swallowed by `|| true`

One step in the pipeline rebased the working branch before committing. The rebase step was followed by || true — a common pattern for "this step might fail and that's acceptable."

Except it wasn't acceptable here. When the rebase fails — for example, due to a conflicted index — the step still exits 0. The pipeline continues. The upload happens. The commit lands on a detached HEAD rather than on main. The video looks uploaded. The queue doesn't see it as done. The same video would publish again on the next run.

I wouldn't have caught this from reading the code because my mental model said "the rebase step is there so the branch is current before we commit — it usually succeeds." The || true was there to handle the case where there was nothing to rebase. Codex read it without that context: if the rebase fails for any other reason, the process continues and produces output that looks correct but is committed to unreachable state (from commit ba21585).

Fix: the pre-gate step now aborts the rebase and fails before any upload if the rebase doesn't complete cleanly.

Category 4 — Platform behavior inference vs. explicit checking

The targeted workflow_dispatch guard — which controls which video file a manual trigger can publish — was written as a job-level if: condition:

if: github.event.inputs.file != ''

On a schedule event, github.event.inputs is undefined. What GitHub returns for github.event.inputs.file when inputs is undefined depends on how the platform casts the access — the GitHub Actions context documentation describes the inputs context as available only for workflow_dispatch and workflow_call triggers, so accessing it on a schedule trigger is accessing an undefined context. My mental model of what it returns ("empty") wasn't the same as what the platform does. The step below the guard used a bash test: [ -n "${INPUT_FILE:-}" ]. That pattern was already verified in production. Two places checking the same condition, one inferring platform behavior and one testing it explicitly.

A second instance in the same review: yt_last_publish_at parsed uploaded_at from the queue JSON and compared it to datetime.utcnow(). All 89 current files end in Z. A future writer adding a naive timestamp would raise a TypeError on the aware/naive comparison and take the health check down entirely.

Fix: the guard moved from if: to bash, matching the existing pattern. The timestamp now coerces naive values to UTC before comparison (from commit 9f79c39).

The common thread

All eight defects came from the same source: the author's knowledge of how the system is supposed to work prevented seeing cases where it doesn't.

"The rebase step usually succeeds." True. "The monitoring thresholds were correct when I wrote them." Also true. "All timestamps in production end in Z." Still true. None of those thoughts are wrong — they describe how the system works when it works. They make it invisible to the author that anything could go wrong.

A systematic reviewer doesn't have those thoughts. It reads the code and asks: what does this actually do when the rebase exits non-zero? What does GitHub return for github.event.inputs on a schedule trigger? What happens when a naive timestamp arrives? Those are first-principles questions that the author stopped asking once the system was built.

This is the same dynamic I see in pipeline health monitoring with detection lag: the job kept producing a well-formed file, so nobody looked inside it. The output inspection pattern addresses the same failure mode but downstream — catching that outputs are wrong. Systematic code review catches it earlier, before the wrong output is ever produced.

The shelf scanner project shows the same pattern at the hardware level. The most consequential operational problems weren't model accuracy issues — they were assumption violations: scan collisions where cron runs overlapped, SD card exhaustion, and Wi-Fi assumptions in the firstrun hook. Those were found through operational experience on a PoC that had produced 19 scans, not through code review — because there was no code reviewer for a headless Pi in a room with nobody watching. GitHub Actions cron timing bugs follow the same pattern in CI: assumptions about when the job will fire, not whether it runs correctly.

What I'd do differently

Run systematic review whenever the scheduling or cadence of a pipeline changes. Monitoring thresholds, gate exit behavior, and git operation sequences are tuned to a specific deployment rhythm. Changing that rhythm turns previously correct assumptions into silent mistakes — exactly the category that causes failures to go unnoticed for 36 to 113 days.

The two categories cheapest to fix in review rather than in production: external-system behavior inference (what does platform X return when Y?) and state-assumption carryover (what do thresholds set under one operational model mean under a different one?).

The two categories a systematic reviewer won't catch: domain correctness ("is this the right threshold for the right business reason?") and requirements drift ("the pipeline is doing what I coded, but not what I actually need"). Those still require the author's judgment.

The skip-tag patterns that prevent recursive publishes and the pausing-workflow trigger traps are both examples of external-system behavior assumptions that are now explicit guards rather than inferred behavior in the single CI pipeline running across three sites and two YouTube channels. Adding a new guard now feels like a checklist item, not a novel decision. That's the direction worth maintaining.

FAQ

Why didn't unit tests catch these?

Unit tests verify that code does what the author expects when the assumptions hold. They don't test what happens when the git rebase exits non-zero, because mocking git state at that granularity is harder than fixing the || true. The assumption violation — that rebase will succeed — is the thing that needs catching, and unit tests don't challenge assumptions; they execute them.

Would a human reviewer have caught these?

A human reviewer who knows the codebase and its history shares the author's context. "The monitoring thresholds were set when this was a daily pipeline" is information that makes the thresholds look reasonable. A reviewer without that history, or a systematic reviewer that reasons from the code structure rather than from accumulated context, is more likely to ask "what's the longest healthy gap?" as a first-principles question.

How often should you run systematic review on a pipeline?

On structural changes: cadence, schedule, external dependency, or trigger condition changes. These are precisely the events that turn old assumptions into new mistakes. Per-feature changes are lower risk because they add behavior without changing the operational model the rest of the system assumes.

Does this mean the code was badly written?

Eight defects over nine months of production use, all caught before causing observed failures, is a reasonable track record. The point isn't that the code was wrong from the start. The point is that structural changes introduce a class of defect that the author is reliably blind to, and systematic review is a low-cost way to surface it before production does.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

BERT-base-uncased, Electra, and why fill-mask models lead production NLP downloads

MORINAGA — Wed, 09 Sep 2026 10:16:00 +0000

When I sort the model catalog in aiappdex by total downloads, the top of the list looks nothing like the AI discourse I read daily:

Model	Downloads	Pipeline tag
all-MiniLM-L6-v2	246,135,287	sentence-similarity
MS-MARCO cross-encoder	83,934,980	text-ranking
BGE-small-en-v1.5	65,039,030	feature-extraction
BERT-base-uncased	58,556,227	fill-mask
Electra-base-discriminator	56,993,793	—
paraphrase-multilingual-MiniLM-L12	45,263,676	sentence-similarity
BGE-M3	36,725,443	sentence-similarity
Qwen3-0.6B	21,444,854	text-generation

Numbers from apps/ai-tools/src/data/models.json as of this writing.

None of the top six are large language models. None of them generate text in a conversational sense. Number four is a model from 2018.

What fill-mask models are actually used for

fill-mask is the pipeline tag for masked language models like BERT, which are pretrained to predict a hidden token in context. In 2018 this was a breakthrough. In 2026 nobody blogs about it, but BERT-base-uncased still pulls 58.6 million downloads a month.

The actual usage is almost always fine-tuning. You take the pretrained checkpoint, attach a classification head, and train on your labeled dataset — customer intent classification, named entity recognition, content moderation labels. At 110M parameters, BERT-base runs on CPU without GPU infrastructure. Fine-tuning takes hours on a laptop. That's why it's in thousands of production ETL pipelines that nobody writes about.

The 2,910 likes suggest it has genuine fans, not just passive users who followed a tutorial once.

Electra: the discriminator that outperforms BERT-base on classification

Electra-base-discriminator sits at 56.9 million downloads — within rounding distance of BERT-base. Electra uses a different pretraining objective: instead of predicting masked tokens, it trains a discriminator to identify tokens that were replaced by a generator model. This produces representations that are more sample-efficient to train and, at the same parameter count, often score higher on classification benchmarks.

In practice the difference is subtle unless you're on a small dataset where the better pretraining generalizes further. Teams that have tried both tend to pick whichever initialized faster for their task and stick with it. The near-identical download counts suggest the user bases are largely overlapping.

Where text-generation fits in these numbers

Qwen3-0.6B sits at 21.4 million downloads — the highest text-generation entry I see in the top tier. It's a capable small language model, and 21M downloads is not a small number. But it sits below BERT-base by a factor of 2.7x.

Part of that gap is age: BERT has years of integrations, downstream checkpoints, and tutorials that compound the download count. But part is real economics. If the task is classifying 100,000 rows of support tickets into 12 intent categories, fine-tuned BERT on a CPU node is still cheaper and faster to serve than a generative model. The 21M downloads for Qwen3-0.6B reflect genuine adoption of small text-generation; the 58M for BERT reflect a different use case that generative models don't displace.

The embedding models above everything else

allMiniLM-L6-v2 at 246M downloads has no close competition. It's a sentence-similarity model from the sentence-transformers library that hit a minimum-viable size-quality tradeoff before anything else did — small enough to run on CPU, good enough to beat BM25 on most semantic search benchmarks.

BGE-small-en-v1.5 (65M downloads) and BGE-M3 (36.7M downloads) are the main challengers on quality, with M3 adding multilingual support and longer context. Switching costs and existing integrations keep allMiniLM-L6-v2 dominant even as newer models exceed it on MTEB benchmarks.

These embedding models are different from fill-mask models in one key way: you use them directly, not as a fine-tuning base. Give them a sentence, get a vector. That makes the integration simpler and the download curve steeper.

What this tells you about production NLP

The top download chart is a lagging indicator of what's deployed at scale, not what's experimentally interesting. Fill-mask and sentence-similarity models show up in this chart because teams found them worth integrating at production volume — not because they're generating benchmark coverage on leaderboards.

If you're building an NLP system and haven't compared fine-tuned BERT against API-based inference for your classification task, the 58 million downloads are a prompt to check. That count didn't accumulate by accident.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Why I'm betting on detection lag over uptime for automated pipeline health

MORINAGA — Wed, 09 Sep 2026 10:15:56 +0000

The problem with "zero errors"

I run several automated pipelines: a daily article publisher, a YouTube video generator, an ETL that pulls from HuggingFace and Steam — all of them on GitHub Actions schedule triggers. For a long time I treated green CI as the signal that things were working. A job that completes with exit code 0 is a healthy job, right?

Not quite. Over the past several months I've been building a pdca-baseline — a document tracking every known failure in this project, with the commit that introduced it and the commit that found it. When I totaled up the numbers, five separate failures had run undetected for weeks or months:

Failure	Days undetected
Steam price field returned the discounted price; every caller quoted it as list price	113
A Reddit source 403'd; `.catch(() => null)` turned it into an empty array	94
97 consecutive uploads failed the same distribution gate	109
Articles published for ~3 months with zero measured human readers	91
A hardcoded "~20x" multiplier, measured once after a data anomaly, never recomputed	36

The common thread: the job kept producing a well-formed file. An empty array is valid JSON. A 403 that returns null is swallowed by the catch block. A green workflow icon does not mean content shipped.

I was tracking uptime (100%) and error rate (0%) the whole time.

What detection lag actually measures

Detection lag is the number of days between the commit that introduced a fault and the commit that found it. You can calculate it from git log without any monitoring infrastructure — just find the two commits and subtract.

The interesting property of this metric is that it rewards a different kind of engineering caution. Uptime pushes you to keep the job running, even if what it's producing is wrong. Detection lag pushes you to check the output, not just the process.

The fault that went 113 days undetected wasn't a crash. The Steam price scraper kept returning valid JSON with a fresh timestamp and the right field names. The value was wrong — it was silently reading the discounted price instead of the list price — but nothing in the pipeline validated values, only structure. Green exit code, wrong data, for nearly four months.

The 91-day case was similar in shape. Articles were being published, cross-posting was working, the pipeline was "healthy." Zero actual humans were reading the output for three months. That failure doesn't show up in any process metric because the process was working. The process was delivering content to a broken distribution path.

This is the failure mode that three approaches to silent failure detection tries to address — but naming the metric gives you something concrete to track over time, not just fix in one-offs.

The counter-example that sharpened the argument

One failure was caught in approximately two days. A fabricated number made it into an article draft — a claim I had no commit-backed evidence for. The independent review step, reading the staged file, flagged it as an unverified metric. Found and removed in two days.

That's the contrast that matters. A wrong sentence in a markdown draft got caught in two days. A wrong number in a database field ran for 113 days unchallenged.

The difference wasn't monitoring sophistication. The code-review step was reading the output and asking "does this claim have evidence?" — not watching a process and asking "did the job complete?" I've written about this distinction in Why I'm betting output inspection beats process monitoring, but detection lag is the measurable version of that observation. It's what you get when you run the experiment on your own history.

The pattern also connects to what happened with survivorship bias in my analytics: videos I manually deleted stayed out of the sample, making the channel look like it was improving. The pipeline was running. The output was wrong in a subtle way. The full account is in How I fixed survivorship bias in my YouTube analytics — that one took a systematic audit, not an alert, to catch.

The falsifiable bet

Here's the version I can be held to: for solo-developer automation pipelines that produce files rather than expose APIs, detection lag measured from commit history will average more than 30 days across all faults found in a six-month window.

The mechanism behind that prediction: pipelines that write files have no live user traffic to surface wrong values. Nobody sends a 400 error when a JSON file contains the wrong price. Nobody complains when an article is published to an empty audience. The feedback loop is severed at the output boundary.

If that's true, the monitoring question changes. It isn't "did the job fail?" — that question has a real-time answer you can alert on. It's "how long before someone noticed?" — and that question requires reading what the pipeline actually wrote.

One cheap heuristic that doesn't require code: if a file that's supposed to change daily hasn't changed in five days, that's worth a look. That single check would have caught at least two of the five failures above. I've been applying a version of this with git-based diffing of weekly data files.

An edge AI parallel: designing for a known detection lag

The shelf-scanning system I'm building on a Raspberry Pi 3 runs at a median of 8.5 seconds per inference scan (n=19 real scans; range 8.4–11.8 s). On a Pi 3, that's honest — you don't get fast inference without hardware that costs more.

The pipeline design acknowledges it. The system runs hourly, and it doesn't confirm a detection until the same gap appears in at least 2 of the last 3 scans — a temporal majority vote. That means the maximum detection lag for a real gap is three scan cycles, or three hours. False positive rate drops because a camera shake or bad frame doesn't trip the alert.

The tradeoff is explicit: three hours to confirm an empty shelf is the chosen bound. The 8.5 s inference time is acceptable because the cadence is hourly, not real-time. I wrote more about the post-processing design in Three layers that made the shelf detector useful on a Pi 3.

That's a better posture than my publishing pipeline had. The shelf detector has a named detection lag bound. My ETL pipeline had an implicit detection lag of whatever I happened to notice, which turned out to be 113 days.

The detection accuracy on the held-out test set is mAP50 0.844. That sounds solid, but it's the model accuracy — the detection lag is a property of the scheduling and voting logic, not the model. You can have a 0.99 mAP model that still takes three months to report a fault if nobody checks the output.

What would change my mind

I'd update this bet if I found a practical process-monitoring setup that reliably catches silent output failures without reading the output. I haven't found one yet — to know that a JSON file contains wrong values, you have to read and validate the values, which means output checking. But if Datadog or Grafana or a similar platform has a mode that samples output content and checks semantic validity, not just "did the file update?", I'd look at it seriously.

I'd also revise if the cost of output sampling grew high enough to avoid. Right now the checks are cheap: was the file updated? Is the array non-empty? Do the values pass basic plausibility tests (price > 0, title length > 0)? Those are cheap enough that skipping them is hard to defend. If the validation needed to catch failures became expensive, that would shift the math.

The third scenario: if pipeline architecture improved to make silent failures structurally impossible — say, by making every write a validated, schema-checked transaction — detection lag becomes less interesting because the failure mode is designed away. I'd take that as a win. The metric matters because the failure mode currently exists, not as an end in itself.

Measuring it going forward

I'm treating detection lag as the primary lagging indicator of pipeline health quality across this project. Each time a new failure is documented in the pdca-baseline, I record fault-introduction commit, fault-detection commit, days elapsed. If that number trends down over time, the output-checking strategy is working. The pdca-prediction patterns analysis has the earlier framing of this — detection lag is the number I'm watching now.

The number I'd most like to report in 30 days: whether the five cases above are outliers or representative for this class of pipeline. I don't know yet. The data will tell.

FAQ

How do you calculate detection lag if git history doesn't perfectly align with deploy timestamps?

I use commit timestamps, not deploy times. Fault-introduction is the commit that introduced the bug; detection is the commit that documents finding it or the first fix commit. For this project those are usually the same event. It's accurate to within a day or two — close enough to be useful as a trend metric.

What about failures caught and fixed immediately?

Those count, and they're the healthy signal. An average detection lag genuinely under 7 days would mean the output-checking strategy is working. I track all detected failures, not just the slow ones.

My pipelines don't have month-long failures. Is the 113-day case an anomaly?

Maybe. My hypothesis is that file-producing pipelines are more susceptible than API-serving ones, because files have no live error feedback from downstream consumers. If your pipeline serves live traffic and users report issues, you probably catch failures in days not months. Mine don't serve live user traffic in that way.

How does this relate to survivorship bias in analytics?

Directly. The deleted-video survivorship bias ran because the analytics pipeline was producing reports that looked plausible. The flaw was only visible when comparing the pre-deletion and post-deletion samples. That's output validation — comparing what the pipeline said to a ground truth the process logs couldn't see. Same mechanism as the detection-lag problem.

Can you prevent this with better schema validation?

Partially. Schema validation catches structural errors — missing fields, wrong types, empty arrays. It doesn't catch semantic errors. The Steam price was the right type (number), within a plausible range, just sourced from the wrong field. Catching that requires domain-specific output validation. Schema is a floor, not a ceiling.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Two times an external API overruled my own repository

MORINAGA — Tue, 08 Sep 2026 10:13:20 +0000

Most automated verification tools assume the repository is the source of truth. If the code says X, X is true. If the commit history documents Y, Y happened. It's a reasonable default — until it isn't.

Two failures I hit this year had the same structure: an automated pass reasoned confidently from the repo, produced a well-formed answer, and was wrong because the actual state lived somewhere the tool couldn't see.

Case 1: the newsletter that didn't exist

While correcting a published article, an automated pass flagged a line claiming there was an active manual newsletter workflow — the article described sending posts by hand to a specific email platform.

The checker read the article, searched the repo for workflow evidence, found nothing committed, and marked the claim unverifiable. That was the right call for the wrong reason. The repo had nothing because the newsletter platform had nothing.

One API call settled it: get the publication, list its posts. The response came back with zero posts and zero drafts. Not "posts I couldn't find in the repo" — zero posts on the platform itself. An active manual workflow is impossible when there is nothing to send. The article was wrong about the mechanism; the API told me the mechanism had never been used at all.

The repo couldn't have told me that. The repo can document what was committed; it cannot document the absence of activity on an external platform. Those are different things, and treating them the same is how the automated pass stayed confidently wrong for a while.

Case 2: the rejection count that kept changing

A different article mentioned four AdSense rejections. The number came from the repo's documented history: four entries, each timestamped.

An automated fact-check pass later changed the article to seven rejections, sourced from the author's own records kept outside the repo. A second automated pass changed it back to four, re-reading the repository.

The actual count is seven. Rejections five, six, and seven arrived as verbal notifications — emails read but never committed. The repo's four was accurate for what was committed; it understated the real count by three.

Two automated passes disagreed with each other not because of a bug but because they each read different sources, both internally consistent, with no signal indicating which was authoritative. The repo said four and was internally consistent. The external records said seven and were internally consistent. The only way to resolve the conflict was to check which source had the more complete view — and the answer was "neither alone."

The fix was mechanical: commit the three missing entries. The harder question is the design one underneath it: when is a repository entry complete, and when is it a documented subset of a larger set?

For this project, the honest answer is that commit history is the subset. Things happen that don't generate commits — API calls that fail silently and get noticed weeks later, verbal information that comes in during a meeting, external systems that change state without notifying the repo. A tool that only reads the repo will confidently "correct" true statements into false ones whenever the truth lives somewhere the repo cannot see.

The same pattern on a Raspberry Pi

The shelf scanner I run on a Raspberry Pi runs a scan every hour and writes the result to a JSON log on the Pi's local filesystem. That log then syncs to cloud storage.

Whether the scan actually ran is not in the GitHub repository. CI has a record of each firmware deploy — that's the repo's version of events. But a green deploy commit is evidence the software was updated, not that the scan executed. The freshness of the scan history lives in the Pi's own output, not in git.

When I built a check for stale scan history — "has the scanner produced a result in the last two hours?" — I had to read the Pi's actual output file rather than querying git log. The repo couldn't answer that question. The scan log could.

This is the same structure as the newsletter and rejection cases: the authoritative source for a fact depends on what the fact is, not on where the code that produces it lives.

What changed in practice

For the newsletter: the article was corrected to remove the manual-workflow claim. One API call, one correction, done.

For the rejections: three entries were committed to the repo. The article now reflects seven. The automated checker no longer contradicts itself across passes.

The broader fix was a distinction in the pipeline between two different failure modes:

"Claim not found in repo" — this means: look externally before marking it false. The repo doesn't hold everything.
"Claim contradicted by repo" — this is a real flag. If the repo's own evidence contradicts a claim, that needs investigation.

Treating these identically was the actual bug. Output inspection — reading what the pipeline actually produced — is different from process inspection, and this is a case where the output lives outside the repo entirely. The repo is a very good source of truth for code and deliberate commits. It's a poor source of truth for anything that was never committed.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

DEV Community: MORINAGA

How three automated writers confused our content-publish gate with one shared label

What the gate checks

The three generators

What the relabeling script found

What this looked like from the output side

The fix is obvious in hindsight

Three ways a GitHub ETL can silently delete valid alternatives — and how I fixed each

Failure 1: An incomplete keep-list from one failed fetch

Failure 2: Seed casing vs GitHub canonical naming

Failure 3: Unguarded bulk delete when the seed file shrinks

The shared principle

FAQ

Four production models I'm watching: speaker diarization, reranking, OCR, prompt safety

pyannote/speaker-diarization-3.1 — 9.26 million downloads, 3,350 likes

BAAI/bge-reranker-v2-m3 — 17.58 million downloads, 1,161 likes

ZhipuAI/GLM-OCR — 2 million downloads, 2,011 likes

Meta/Prompt-Guard-86M — 4.37 million downloads, 397 likes

Three YouTube Data API v3 behaviors that broke my analytics classifier

1. Channel handle lookup fails silently when the handle doesn't match

2. View counts require a second API call

3. Including "unknown" videos corrupts the archetype ranking

Five things I noticed this week: deletion bias, hidden assumptions, production AI

1. I deleted my way to a fake improvement in YouTube analytics

2. Cadence assumptions are the sneakiest hidden state in GitHub Actions

3. The headless Pi scanner hit the same class of implicit constraint

4. Production HuggingFace downloads are dominated by models nobody blogs about

5. Detection lag tells you more than uptime ever did

Three Signals from 284 Published Articles That Changed How I Plan New Content

Signal 1: Recent articles outperform older ones, despite less time to accumulate

Signal 2: Weekly-digest format has the highest median reactions

Signal 3: Concentration is extreme — two articles account for a disproportionate share

What I'm not changing

What I Learned About Declaring Source-of-Truth Per Claim in an Automated Pipeline

The class of failure: confident wrong corrections

Two cases from my pipeline

What per-claim source declaration looks like

The maintenance overhead question

Where implicit mapping stays acceptable

What I would do differently

FAQ

Chronos-2, CLIP, and MobileNetV3: three production AI models for non-text workloads

Chronos-2 — zero-shot time-series at 25.7M downloads

CLIP-vit-base-patch32 — 19.9M downloads for cross-modal search

MobileNetV3-small — 17.4M downloads for edge classification

The pattern across all three

How systematic review catches the GitHub Actions assumptions authors can't see

The pipeline and why the cadence changed

Category 1 — Monitoring thresholds tuned to the wrong cadence

Category 2 — A rejected spec consuming a publish slot

Category 3 — Git state swallowed by || true

Category 4 — Platform behavior inference vs. explicit checking

The common thread

What I'd do differently

FAQ

BERT-base-uncased, Electra, and why fill-mask models lead production NLP downloads

What fill-mask models are actually used for

Electra: the discriminator that outperforms BERT-base on classification

Where text-generation fits in these numbers

The embedding models above everything else

What this tells you about production NLP

Why I'm betting on detection lag over uptime for automated pipeline health

The problem with "zero errors"

What detection lag actually measures

The counter-example that sharpened the argument

The falsifiable bet

An edge AI parallel: designing for a known detection lag

What would change my mind

Measuring it going forward

FAQ

Two times an external API overruled my own repository

Case 1: the newsletter that didn't exist

Case 2: the rejection count that kept changing

The same pattern on a Raspberry Pi

What changed in practice

Category 3 — Git state swallowed by `|| true`