DEV Community: MORINAGA

Four filters I apply when pulling HuggingFace models into an AI tools directory

MORINAGA — Wed, 01 Jul 2026 09:26:17 +0000

HuggingFace's model hub has over 900,000 models as of mid-2026. Surfacing all of them on aiappdex.com would produce noise, not a directory. The ETL that runs nightly to update the AI tools directory applies four filters before any model is considered for inclusion. Here's what each filter does, what it catches that the previous filter missed, and what the chain still doesn't solve.

Filter 1: pipeline tag — only end-user-facing tasks

The HuggingFace API returns a pipeline_tag field on every model. The allowed set in my ETL is not the full HuggingFace taxonomy:

text-generation, text-classification, token-classification,
question-answering, summarization, translation, image-classification,
image-generation, image-to-text, text-to-image, automatic-speech-recognition,
text-to-speech, audio-classification, zero-shot-classification,
feature-extraction, sentence-similarity

What this excludes: image-segmentation, object-detection, depth-estimation, tabular-regression, and a dozen other computer-vision pipelines that are real ML tasks but not tools most directory visitors would search for. Also excluded: models with no pipeline_tag at all, which covers roughly 40% of the hub — mostly adapter weights, partial checkpoints, and fine-tune datasets uploaded alongside a model rather than the model itself.

This filter alone cuts the candidate set by roughly 80%. The resulting pool is still large enough to be useful — text-generation alone has hundreds of thousands of models — but it's a tractable number.

Filter 2: minimum likes threshold — filtering out abandoned and test uploads

Every model on HuggingFace has a likes count. The current threshold in the ETL is 30. Models below that get skipped entirely.

Thirty likes is a very low bar. What it actually catches: test uploads that were never intended for public use, deprecated model versions that were superseded by a renamed upload, and fine-tunes of popular base models trained on private datasets and uploaded without cleanup. These aren't useful directory entries — they don't have documentation, they often have incorrect or placeholder metadata, and they frequently return 404 on the download endpoint even though the API record exists.

The 30-like threshold isn't magic. It's the point where I stopped finding entries that were clearly accidental public uploads. Ten likes still produced a lot of noise; 30 produced much less. The threshold is a config value I can change per pipeline_tag if needed — text-generation models benefit from a higher threshold (I'd probably raise it to 50 for that pipeline if I wanted stricter quality) while rarer pipelines like text-to-speech work better with a lower cutoff because the ecosystem is smaller.

Filter 3: last-modified recency — flagging dormant models

This filter doesn't exclude models; it sets a low_activity flag on models that haven't been modified in 14 months. That flag gets stored in the Turso database and surfaced in the directory as a "last active" label rather than a hidden exclusion.

Why flag instead of exclude? Because an old model isn't necessarily a useless model. GPT-J 6B is from 2021 and still appears in production stacks. BERT-base-uncased is from 2019 and is used in half the fine-tuning tutorials published in 2026. Excluding by recency would misrepresent the landscape.

What the low-activity flag does catch: models that were announced with fanfare, got early likes, and then quietly went dormant when the authors moved to a new architecture. Without this flag, those models appear alongside actively maintained alternatives without any signal that maintenance stopped. For someone evaluating tools for production use, that distinction matters.

The 14-month threshold aligns with how HuggingFace itself handles model cards — a model page that hasn't been touched in 14 months almost certainly doesn't reflect the current state of the codebase or the upstream library it depends on.

Filter 4: gated and private models — inclusion requires public weights

Models marked gated: true on HuggingFace require login and a request form to download. Models marked private: true aren't downloadable at all. Neither appears in the directory.

The rationale isn't philosophical — it's practical. A directory entry for a model that visitors can't access without a form submission is a poor UX. The directory's value is "find a model, go use it." Gated models break that flow entirely for anyone who hasn't already been approved.

This filter has one real cost: it excludes some important models. Meta's Llama 3 series launched gated, as did several Mistral fine-tunes during their initial access period. The directory missed those for the window between public announcement and when gating was lifted. That's a real gap. The three-tier content quality ladder that handles model content upgrades would apply here too — if I wanted to include gated models with a "requires access request" label, I could add a tier for that. I've chosen not to for now because the "access request required" experience isn't what the directory is designed around.

What the chain still doesn't solve

These four filters produce a candidate set that's tractable and skewed toward useful entries. What they don't catch:

Duplicate fine-tunes. There are thousands of Llama-3.1-8B fine-tunes on HuggingFace, all passing the pipeline tag filter, all with enough likes, all public and actively maintained. The directory clusters them by base model but doesn't deduplicate in a meaningful way. Someone searching for an instruction-following model still faces a wall of variants.

Quality of the model card. A model that passes all four filters might have a model card that says "fine-tuned for [task]" with no further detail — no eval results, no intended use, no known limitations. The ETL can't infer quality from card text reliably. That's what Claude Haiku's editorial generation step handles: a prompted generation that forces structured outputs around audience fit and limitations. But it's worth naming that the ETL filters select for metadata quality, not model quality.

Pricing and deployment complexity. HuggingFace doesn't expose whether a model runs in 8GB of VRAM, requires a dedicated A100, or is practical to call via the Inference API without self-hosting. That data isn't in the API response. It's the kind of structured attribute that would make the directory genuinely useful for someone deciding between models — and it's something I'd want to add as a manual editorial field rather than an ETL-derived one.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Why I'm betting on Claude Code over Cursor for a solo dev pipeline

MORINAGA — Wed, 01 Jul 2026 09:26:12 +0000

I used Cursor as my primary AI coding tool from early 2025 through February 2026. In March, when I started the three-directory-site experiment I've been documenting here, I switched to Claude Code as my main driver. That switch wasn't impulsive — I tried both on the same tasks for about two weeks before committing. Here's the specific bet I'm making, the case against it that I find genuinely compelling, and the three conditions under which I'd reverse the decision.

The bet, stated plainly

By December 2026 — nine months into this project — Claude Code will have saved me more total wall-clock time on this experiment than Cursor would have, net of context-restart overhead and the overhead of not having inline tab completion. The measurement is informal but the conditions aren't vague: if I'm spending more than 20 minutes per week re-establishing context that Cursor would have kept in a sidebar chat history, I'm wrong.

I'm not claiming Claude Code is better across all developer workflows. My bet is specific to the task distribution this project has required: multi-file refactors touching five or more files simultaneously, GitHub Actions debugging where terminal output is the primary signal, automated pipeline scripts that need AI assistance at invocation time, and content-generation runs that consume the staged git diff as input.

What pushed me toward Claude Code

The immediate catalyst was the GitHub Actions CI setup. The three-site architecture runs a nightly ETL, a daily article generation job, a Bluesky queue refill, and several post-deploy checks — each as a separate workflow. Debugging those workflows from Cursor's chat mode is awkward. Cursor sees the YAML file correctly; it doesn't see the runtime logs that show exactly which step failed and what the failing bash command produced. I had to copy error output from the GitHub Actions interface into Cursor's chat manually, which breaks flow.

Claude Code sits in the terminal alongside the git output, the pnpm install errors, the node script stack traces. I can paste a failing run log directly into a session without switching context. That terminal-native loop — observe failure, invoke AI, inspect proposed fix, run command — is where I first noticed a meaningful productivity gap.

The second factor is multi-file coherence. The shared Claude Haiku client is imported by five different ETL scripts across three separate apps. Refactoring it — adding a retry parameter, changing the caching behavior — means touching all five call sites simultaneously. Claude Code can open all five files in context, reason about which call sites need updating and which don't based on usage patterns, and produce a coherent multi-file diff explanation. Cursor's "apply to multiple files" flow surfaces one diff at a time with manual approval at each step. For this specific operation — a cross-repo parameter change — I find Claude Code's approach faster.

Third is the article-generation pipeline itself. The content quality gate runs an audit script on every generated file. The routine I run uses staged git output to feed a reviewer, then optionally patches the article before committing. That whole loop — generate, stage, review, fix, re-stage, commit — runs in the terminal. Claude Code can execute bash commands, inspect what changed, and iterate without clipboard handoffs. In Cursor I'd break that flow every time I needed to check the audit output or run a pnpm script.

What I'm giving up

Cursor's inline edit mode is genuinely better for micro-changes. CMD+K opens a floating edit bar at the cursor position, accepts a one-sentence description, and shows an inline diff that accepts or rejects in under two seconds. Claude Code has no equivalent. If I want to rename a variable or flip a conditional, I describe the location in the terminal, wait for the tool to navigate there, and approve the change — objectively slower.

Tab completion is the other thing I miss. Cursor's completions predict what you're about to type in a familiar codebase with surprising accuracy. In the TypeScript ETL scripts I iterate on constantly, Cursor already knows I'm about to write await db.execute({sql: and completes the pattern including the object shape. Claude Code has no tab-complete mode; it's interactive-only.

The third gap is session continuity. Cursor's sidebar chat persists history across sessions. I can scroll back in a Cursor conversation and see the discussion that explained a design decision two weeks ago. Claude Code starts fresh on every invocation. If I'm debugging something I touched four days ago, I'm re-establishing context from git log and file reads rather than from a conversation thread that already captured the reasoning.

The counterargument I take seriously

The strongest case against my bet: Claude Code's terminal-native strength is also its ceiling. The operations where it beats Cursor — multi-file refactors, CI debugging, pipeline automation — are a minority of actual characters typed in a development session. Line-level edits, variable renames, docstring updates, quick function calls — those are the majority. Cursor handles them faster.

If the correct mental model is "80% of dev time is small edits, 20% is large operations," then optimizing for the 20% with Claude Code while taking a speed penalty on the 80% is a net loss. Cursor, covering 80% well and 20% adequately, might win on total wall-clock time even if Claude Code wins on per-operation speed for the big tasks.

I don't have tracked data to refute this cleanly. My intuition is that this project, specifically, is skewed toward the 20% end — pipeline-wide changes, new ETL integrations, debugging CI failures — more than a typical single-app product build would be. The AI directories bet has the same honest structure: I'm making a claim based on structural reasoning, not on clean measurement.

What would resolve this: tracking dev time by operation type for four weeks and comparing the two categories. That's a straightforward measurement I haven't run. If small edits consume more than two-thirds of my Claude Code sessions, the counterargument wins.

The cost structure

Both tools' monthly costs are close enough that cost alone isn't the deciding factor, but the structure matters for how I think about usage.

Cursor Pro is $20/month flat, covering unlimited completions and a monthly cap on "premium" model uses — Claude Sonnet and GPT-4o, as of this writing — with automatic fallback to a smaller model when the cap is hit. Predictable cost, opaque per-operation consumption.

Claude Code bills against an Anthropic API key directly. For my usage pattern — roughly three to five complex sessions per day on a project at this scale — the monthly API cost lands between $15 and $30 depending on session complexity. The variance comes from how often I ask for full-codebase reads versus targeted edits. It's not consistently cheaper than Cursor Pro, and it's not consistently more expensive.

The meaningful difference is visibility. With Claude Code I can see exactly what each session consumed. With Cursor Pro I don't know whether a "apply to 12 files" operation used one premium credit or ten. For a project where I'm tracking every dollar of infrastructure cost, per-operation visibility changes how I think about usage patterns.

How I partition the workflow in practice

I use Claude Code for anything that starts with a problem statement spanning multiple files: "the ETL is writing duplicate entries — find where the upsert logic lives and figure out why it's firing twice for the same ID." Terminal access plus file-reading plus bash execution in one context is worth the micro-edit tradeoff for that class of problem.

For genuinely small edits — a Tailwind class adjustment, a typo in a component — I open the file directly in VSCode and edit manually. No AI involved. That's faster than either tool for operations with clear, exact solutions.

What I've essentially done is partition my editing: Claude Code for architectural operations, direct editing for surgical fixes, no AI for trivial changes. Cursor was attempting to cover all three categories; the result was friction at both ends because the tool can't optimize simultaneously for "large autonomous operation" and "two-keystroke inline fix."

The static site rendering choice followed similar reasoning: picking one approach for a specific constraint set rather than picking the tool that's most general. I'm applying the same thinking to development tooling.

What would change my mind

Three signals would push me back to Cursor.

Context loss compounds past a threshold. If I find myself spending more than 20 minutes per week re-explaining architectural decisions to fresh Claude Code sessions that should have retained them, the continuity gap stops being an acceptable tradeoff. That threshold is specific enough to evaluate month-by-month.

Cursor ships terminal-native agentic mode. Cursor is actively developing agentic capabilities. If they ship a mode where Cursor executes terminal commands, observes output, and iterates without requiring IDE focus — essentially what Claude Code's bash tool does — the workflow gap I've described narrows to near zero. I'd run a direct comparison again at that point.

Task distribution shifts toward UI iteration. This project is currently infrastructure-heavy: ETL pipelines, CI workflows, cross-posting automation, the pairwise compare page generation. If it matures into mostly front-end iteration — layout experiments on the directory pages, A/B testing components — the small-edit / tab-completion advantage that Cursor holds would outweigh the pipeline operations advantage. The bet is partly a claim about what the project will continue to require.

The December 2026 checkpoint

The AI directories bet has a formal October 2026 deadline. This tooling bet is softer — I'll check in by December 2026 with whatever the data shows. I'll report: how often I hit the context-loss pain point, whether the task distribution stayed infrastructure-heavy, and whether either tool materially changed its offering. If I've switched back to Cursor by then, I'll say so with specifics.

One thing I won't do is rationalize ambiguous signals optimistically. The same commitment I made about Bluesky automation quality — systematic gates, not self-review — applies here. If the measurement says I'm wrong, I'll say I'm wrong.

FAQ

Can you use Claude Code and Cursor at the same time?

Yes, and I occasionally do. Claude Code sessions happen in a terminal window; Cursor runs in the IDE alongside. The main friction: if both are running Claude Sonnet simultaneously, they're competing for the same API rate limits. In practice I don't hit conflicts on this project's volume, but it's something to watch for longer agentic sessions.

Is Claude Code available everywhere Cursor is?

No. Cursor is a full IDE replacement available on Mac, Windows, and Linux. Claude Code is a CLI that requires a terminal. It doesn't have a native Windows GUI experience as of mid-2026, though WSL2 works. For developers primarily on Windows without WSL, this is a practical blocker.

What about GitHub Copilot — does either replace it?

Cursor includes Copilot-style tab completion with its own model backend; you don't need a separate Copilot subscription if you're using Cursor Pro. Claude Code doesn't offer tab completion. If inline completions are your primary AI coding use, Claude Code isn't a Copilot replacement — it's a different tool that doesn't try to be.

How does this tooling choice affect the pipeline automation?

Directly: the ETL scripts, GitHub Actions YAML, and article generation routines are all authored and debugged through Claude Code. Indirectly: the content quality gate and the QC review runs are shell scripts that fit naturally into a Claude Code session but would require clipboard handoffs in Cursor. The tool shapes how I build the automation, which shapes what automation I'm willing to maintain.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

How I verify affiliate CTAs are actually rendering in production

MORINAGA — Sun, 28 Jun 2026 22:17:37 +0000

I run three directory sites that display affiliate links, AdSense slots, and Amazon blocks — but only when the corresponding environment variables are set in Cloudflare Pages. When the variables aren't deployed, those sections simply don't render. No error. No broken layout. Just missing revenue, invisible unless you check.

This happened twice in the first month. A redeploy would go out without the affiliate env vars being re-applied. The site looked identical to a working version at a glance. Clicking around would eventually reveal the missing CTA, but only if you happened to land on the right page type.

I wrote scripts/check-affiliates.mjs to make that check fast and explicit.

What the script does

The script checks three sites in sequence. For each site, it:

Fetches /sitemap-index.xml to find the sub-sitemap for detail pages
Picks one detail URL from that sitemap
Fetches the raw HTML
Checks for specific strings that indicate each CTA is rendered

The output is a plain pass/fail report per site:

→ aiappdex.com
  ads.txt: ✓ pub ID set
  sample: https://aiappdex.com/models/qwen2-7b/
  affiliate CTA "Run this model on": ✓ rendered
  AdSense slot: ✓ in HTML
  Amazon block: ✗ hidden (PUBLIC_AMAZON_TAG not deployed)

One line per check, one pass per site, one run to confirm three deployments.

The sitemap crawl

Hardcoding a URL would make the check brittle — detail URLs change when slugs are regenerated, and I'd rather not maintain a separate list. Using the sitemap is more robust: it's the canonical URL source the site already generates.

async function pickFirstSlug(site, prefix) {
  const sitemapRes = await fetch(`https://${site}/sitemap-index.xml`);
  if (!sitemapRes.ok) return null;
  const idx = await sitemapRes.text();
  const subSitemap = (idx.match(/<loc>([^<]+)<\/loc>/) ?? [])[1];
  if (!subSitemap) return null;
  const subRes = await fetch(subSitemap);
  if (!subRes.ok) return null;
  const sub = await subRes.text();
  const urls = [...sub.matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]);
  const detail = urls.find((u) => u.includes(prefix) && !u.endsWith(prefix));
  return detail ?? null;
}

It parses <loc> tags from the sitemap XML with a regex rather than a full XML parser. That's intentional — DOMParser isn't available in Node by default, and adding a dependency for sitemap parsing felt disproportionate. The regex works because sitemap XML format is structurally consistent; a more complex format would warrant a parser.

One thing to note: the function takes a prefix argument (like /models/ or /games/). That's how I distinguish detail pages from the index pages that also appear in the sitemap. I want a URL like /models/qwen2-7b/, not /models/.

Checking ads.txt and affiliate strings

The ads.txt check is a separate fetch, not the HTML check. It looks for the AdSense publisher ID pattern:

const adsTxtRes = await fetch(`https://${site}/ads.txt`);
const adsTxt = await adsTxtRes.text();
const hasAdsensePub = /pub-\d{10,}/.test(adsTxt);

The HTML checks are string presence checks against the fetched page:

const hasSection = html.includes(section);
const hasAdsense = html.includes("adsbygoogle") && html.includes("data-ad-client");
const hasAmazon = html.includes("Gear up on Amazon") || html.includes("amazon.com/s?k=");

The section variable is site-specific. For aiappdex.com it's "Run this model on"; for findindiegame.com it's "Find on other stores"; for ossfind.com it's "Self-host on". These are heading strings that only appear in the rendered HTML when the relevant env var is set.

I deliberately check strings that are human-readable rather than env var names or data-attributes. If the heading renders, the CTA is live. If the heading is absent, something upstream didn't connect. The message in that case tells me exactly which env var to check in Cloudflare.

Why this is better than a visual check

The pattern I was relying on before — opening a few pages after a deploy and eyeballing them — has two problems. First, it's slow across three sites with multiple CTA types. Second, it's unreliable at catching conditional rendering: an affiliate block that's absent looks the same as a block I intentionally disabled or that I haven't scrolled to yet.

A script that fetches programmatically, checks presence by string match, and reports pass/fail for each CTA type takes about two seconds and catches the failure unambiguously. The output is readable in a terminal and doesn't require loading a browser.

This connects to the same principle behind the three-tier content quality ladder: checks at different stages catch different things. Post-deploy verification catches deploy-time configuration problems. Pre-commit linting catches content problems. Neither replaces the other.

What I'd add

Right now the script only checks one sample page per site. A more thorough version would check one page from each content type per site — a model page, a compare page, an alternatives page — since some CTAs only render on specific page types. That would require more sitemap traversal but would catch more edge cases.

The output format is human-readable but not machine-parseable. If I wanted to hook this into CI and fail a deploy when a CTA is missing, I'd add a JSON output mode and return a non-zero exit code on any ✗. For now I run it manually after deploys — it takes less than ten seconds and the terminal output is enough.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

How I built a content quality gate that stops bad articles before they publish

MORINAGA — Sun, 28 Jun 2026 22:17:32 +0000

I run three directory sites and a content pipeline that generates and cross-posts articles to Dev.to, Hashnode, and Bluesky automatically. The pipeline has been running for about six weeks. Early on I found a category of failure that no amount of CI infrastructure was catching: content quality problems. Wrong tags. Cliché phrases that slipped past self-review. Articles that implied specific traffic metrics I couldn't back up. Fabricated specificity disguised as honest reporting.

The solution was scripts/audit-articles.mjs — a lint-style quality gate that runs on every new article before the publish step. It works the way eslint works for code: structured checks, clear error/warning distinction, strict mode for pre-publish, lenient mode for a historical baseline.

Why a lint gate and not a manual review

The specific failure mode I was trying to prevent was this: automated generation leaves an article that reads fine on first pass but fails on systematic inspection. A cliché phrase at the start of a section. The tag "seo" slipping in when the pool explicitly forbids it. A word count of 580 when the spec requires 600-900 for a lightweight article. These aren't hard to catch — they're tedious to catch every single time, and tedium is where manual review degrades.

The pre-post Bluesky QC gate I built earlier applies the same principle to social posts: systematic gates catch what self-review misses reliably. For articles, the gate runs before the publish workflow can touch the file.

The JSON-LD audit script runs a similar check post-deploy against live pages. audit-articles.mjs runs pre-commit, against local markdown. Catching it before it ships is always cheaper than chasing it down after.

What the gate checks

The script runs about 12 distinct checks per file, split into errors (fail hard in strict mode) and warnings (report but continue by default).

Frontmatter structure. Four required keys must be present: title, description, tags, publish_to. Missing any of them is an error. publish_to must contain only known targets. tags must be an array of exactly 4 items, each drawn from an explicit pool of 18 allowed values:

const TAG_POOL = new Set([
  "ai", "astro", "webdev", "showdev", "typescript", "javascript",
  "indiehackers", "productivity", "opensource", "programming", "tutorial",
  "machinelearning", "claude", "anthropic", "vercel", "turso", "tailwindcss",
  "githubactions",
]);

Any tag outside the pool is an error, not a warning. The pool exists to prevent both "seo" (explicitly prohibited in the spec) and organic drift toward one-off tags that don't build topic coherence. The editorial layer over programmatic content has a similar constraint: structure imposed at ingestion time is easier to maintain than structure enforced by convention.

Title and description length. Titles over 90 characters are an error — Dev.to and Hashnode truncate feed titles beyond that. Descriptions over 200 characters are an error — the meta description budget for Google and Hashnode display.

Word count. This is a warning rather than a hard error because the threshold varies by article archetype. A lightweight at 580 words isn't technically failing a hard constraint, but the flag forces a decision: add a section, or accept the shorter count and justify it. The word count always appears in the summary line regardless, so there's no hiding from it.

Cliché detection. 14 literal phrases are checked case-insensitively against the full body. The list covers diving metaphors, fast-paced-world openers, hyperbolic tech superlatives, and in-this-article throat-clearing — the standard set of AI-assisted writing tics. I won't reproduce all 14 here for an ironic reason I'll cover in "What I'd do differently."

The check is simple String.prototype.includes, not regex, because the phrases are specific enough that false positives aren't a real concern. An article using the typical "world of AI" framing is probably using clichéd structure; the check forcing me to see it is the point.

Fabricated metric detection. Two regex patterns catch the most common forms of fabricated social proof:

const FABRICATED_METRIC_PATTERNS = [
  /\b(\d{2,3},?\d{3,}|\d{4,})\s+(visit|view|readers?|users?|subscribers?)\b/gi,
  /\branked #1\b/gi,
  /\bmillion(?:s)?\s+of\s+(?:users?|readers?|developers?)/gi,
];

The first pattern catches round numbers followed by social-proof nouns like "visitors", "users", or "subscribers" — numbers presented as fact when the sites are young enough that no credible claim to that kind of traffic exists. The pattern requires a four-digit-or-higher number followed directly by a social-proof noun. It catches claimed traffic figures but not numeric references in code or data discussion.

This check produces an error, not a warning, because fabricated metrics are the one category I genuinely can't allow. The whole premise of the three sites experiment depends on readers trusting that I'm reporting honestly. One fabricated stat poisons that trust retroactively.

Internal and external link presence. For pillar-sized articles (over 1200 words), the check warns if internal links number fewer than 5 — the spec requires 5-10 for pillar archetype. For any article over 600 words, it warns if there are zero external links. The check uses a simple regex:

const internalLinks = (body.match(/\]\(\/[^)]+\)/g) || []).length;
if (wc > 1200 && internalLinks < 5) {
  warnings.push(
    `pillar-sized (${wc} words) has ${internalLinks} internal links — spec requires 5-10`
  );
}

The noindex gate for programmatic pages uses a similar "catch before ship" logic at the infrastructure level. The article gate applies it at the content level.

Sentence repetition. Any sentence appearing three or more times in the same article generates a warning. This catches a specific AI-assisted writing failure: a paragraph occasionally gets reformulated twice, and the reformulation ends up identical to the original a few hundred words later. The check normalizes to lowercase and trims whitespace before comparing.

Strict versus lenient mode

The gate has two distinct behaviors depending on how it's called. The Node.js process.exitCode documentation is the relevant primitive — returning 1 from main() is enough to block any downstream step that checks exit codes.

When run against a single file (node scripts/audit-articles.mjs path/to/article.md) or with --strict, errors cause a non-zero exit code and the publish step stops.

When run against all articles without explicit paths (node scripts/audit-articles.mjs), errors are reported but don't fail the process. This baseline scan mode exists because older articles pre-date some current rules, and a retroactive constraint would block everything. The three-tier content quality ladder applies here: different content tiers have different quality expectations.

The article generation routine runs strict mode on each newly staged file before committing. The publish workflow runs against the specific file being published. The all-articles scan runs periodically to report on historical drift without failing anything.

Title duplicate detection across the repo

When scanning all articles (not single-file mode), the gate runs a cross-article deduplication check. Titles are normalized to lowercase alphanumeric, whitespace collapsed, then compared:

function detectTitleDuplicates(reports) {
  const titles = new Map();
  for (const [path, r] of reports) {
    if (!r.meta?.title) continue;
    const key = r.meta.title.toLowerCase().replace(/[^a-z0-9]+/g, " ").trim();
    const existing = titles.get(key) ?? [];
    existing.push(path);
    titles.set(key, existing);
  }
  return [...titles.entries()]
    .filter(([, paths]) => paths.length > 1)
    .map(([key, paths]) => ({ key, paths }));
}

This catches rephrase cases more than exact duplicates (which would be obvious). "Why I use Turso for my Astro monorepo" and "Using Turso libSQL in an Astro monorepo" normalize to similar-enough strings to surface as candidates. The check doesn't do semantic similarity — it doesn't need to. Structural overlap is enough of a signal to warrant a second look, given how the pipeline-aware content variants approach works.

What actually got caught

Since adding the gate, here are the real categories it's flagged that I fixed before publishing:

Tag outside pool: twice. Both times were "seo" appearing in generated frontmatter. Reliable pattern: when generating frontmatter from a prompt, the model includes obviously relevant tags that are explicitly off-limits in the spec.

Word count too low: three lightweight articles came in at 540-580 words. Two got an additional section and reached spec. One I accepted shorter — the content was complete and adding words would have been padding.

Fabricated metric match: once. The generated draft said "early testers report" something in a way that parsed as a quantified claim. Fix was removing the number.

Missing external link: regularly. Generated articles sometimes make claims about tools or APIs without citing the primary source. About half the time I add a link on review; the other half I rephrase to not make a claim that requires one.

What I'd do differently

The cliché list is static. A smarter approach: add phrases from real failures after the fact. When a specific phrase slips through and I notice it post-publish, add it. The list has grown from 8 to 14 entries through manual additions, but there's no automated feedback loop. One possible improvement: run the gate on published articles periodically and flag new patterns.

The biggest gap: the checker doesn't strip code blocks before running cliché or metric checks. This article itself demonstrates the problem — I had to describe the cliché list in prose rather than including it as a JavaScript literal, because reproducing the exact phrases inside a code block triggers the gate. The fix is straightforward: split the body on triple-backtick fences, check only the prose regions. A two-pass preprocessing step before any pattern check would resolve this entirely. I haven't shipped it yet because the current behavior just means I write around it, which isn't painful enough to force the fix.

CI integration is partial. The gate runs as part of the article generation routine and as a verify step, but it doesn't run as a blocking step in the publish workflow itself. That means a file committed outside the routine could bypass the gate entirely. The fix is one line in publish-articles.yml — a step I haven't prioritized because the routine path is currently the only real publishing path.

The EEAT transparency pages I built approach content credibility from the site structure side. The lint gate approaches it from the individual article side. Both serve the same goal: content that doesn't embarrass the project in hindsight.

FAQ

Why not use an existing prose linter like alex or write-good?

alex focuses on inclusive language; write-good checks passive voice and weak qualifiers. Neither checks frontmatter structure, tag pools, or fabricated metrics — the pipeline-specific failures I actually need to catch. A domain-specific gate catches domain-specific failures better than a general-purpose tool.

How does the fabricated metric check handle code blocks?

Currently it doesn't exclude them, which is the known gap. Numeric references in code examples occasionally trigger false positives. I review the flagged line location before treating it as a real failure. Pre-processing the body to strip triple-backtick blocks would fix this.

Why warn on internal link count instead of hard-erroring?

Because article type isn't always deterministic from content alone. A 1200-word article might be a pillar or a borderline lightweight. Treating it as a warning preserves the ability to make a judgment call; treating it as a hard error would block edge cases that are genuinely fine.

What happens to historical articles that fail current checks?

Baseline scan mode reports them without failing. Historical content is in a different tier. New articles must pass strict mode; historical articles are known issues reported for visibility, not blocked.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Two-host AI dialogue specs: how I structure YouTube longform scripts with A/B speaker JSON

MORINAGA — Sun, 28 Jun 2026 08:39:02 +0000

The video pipeline I've been building for two YouTube channels running off this monorepo started with short-form vertical clips — a single narrator, a single slide, done. Longform is different. A ten-minute explainer with one voice and no conversational variation is hard to watch even when the content is good. I wanted something that felt like two people talking through a problem, not a text-to-speech audiobook.

The solution was a two-host dialogue spec: a JSON file where each line of audio is tagged with a speaker (A or B), and the build script renders it as a full-length video alternating between two neural voices.

What the spec looks like

The simplest possible spec:

{
  "title": "How Turso libSQL compares to Cloudflare D1",
  "description": "...",
  "tags": ["database", "cloudflare", "turso"],
  "privacy": "public",
  "segments": [
    {
      "speaker": "A",
      "text": "Turso and D1 look similar from the outside — both are SQLite-compatible edge databases.",
      "slide": { "kind": "title", "heading": "Turso vs D1" }
    },
    {
      "speaker": "B",
      "text": "Right. But they differ significantly on branching, replication topology, and cost model once you scale past the free tier."
    },
    {
      "speaker": "A",
      "text": "Let's go through each. First, branching.",
      "slide": { "kind": "section", "heading": "Branching" }
    }
  ]
}

A few things to notice. The slide field is optional — when omitted, the build script holds the previous slide while the audio plays. This means you don't need a new visual for every sentence, which would be exhausting to maintain and would produce a choppy video. A new slide appears only when there's something worth showing.

The speaker field maps to a voice. In build_longform.py:

VOICE = {
    "A": os.environ.get("LF_VOICE_A", "en-US-GuyNeural"),
    "B": os.environ.get("LF_VOICE_B", "en-US-AvaNeural"),
}

Both are edge-tts neural voices — a Python wrapper around Microsoft Edge's text-to-speech API that gives access to the same neural voices as the browser without requiring an Azure subscription. The A/B assignment came from testing: one lower/measured voice for exposition, one that sounds more conversational for follow-up and counterpoint. You can override both with environment variables, which matters if the default voices aren't available in a given edge-tts catalog version.

How the build works

build_longform.py processes the spec linearly:

For each segment with a slide, render the slide to PNG via slides.py
Synthesize the segment's text with edge-tts for the assigned speaker voice, writing to an mp3
Build a silent video clip from the PNG, then mux it with the audio
After all segments: concatenate all clips with ffmpeg

The result is a single output.mp4 where each visual change happens exactly when a new slide is specified in the spec — usually at section transitions, not on every sentence.

If a segment has no slide key, the previous slide's PNG is reused. The timing automatically matches the audio duration because each clip is built from its own audio file. No manual timestamp editing.

The CI step that runs this writes the output path to an environment file:

YT_OUTPUT_PATH=/tmp/longform/output.mp4

The downstream YouTube publish step reads that variable and uploads. Same pattern as the short-form video pipeline I wrote about.

What the spec generator produces

The specs aren't written by hand. A Claude call takes a topic and an outline and produces the full segment list, deciding where slides should appear, which speaker handles which part of the argument, and what heading text goes on each slide.

The prompt instructs the model to split responsibilities clearly: speaker A leads and introduces, speaker B challenges, adds nuance, or extends with examples. This produces a conversational dynamic that's more engaging than a single narrator even though neither voice is a real person.

One thing that took adjustment: Claude tends to generate very even A/B splits — roughly alternating every sentence. Real dialogue isn't that regular. I added an instruction to vary the run lengths: sometimes A speaks three sentences before B responds, sometimes B only adds a single sentence. That small change makes the output feel less mechanical.

What I haven't solved yet

The PNGtuber-style character art mentioned in the build script (_host_assets() function) is asset-gated and returns None currently — I haven't made the visual assets for either host. The code path is there for when I do, but for now the video is slide-only with audio.

The slide renderer (slides.py) is also limited to a few layouts: title cards, section headers, comparison tables, bullet lists. Richer layouts like code blocks with syntax highlighting or real diagrams would require more work in Pillow or a headless browser, which I'm deferring. The Mermaid + matplotlib diagram pipeline I built for short-form videos doesn't cleanly transfer to longform because the timing model is different.

The two-voice format is working for the content I'm producing. Whether it affects watch time versus a single-voice format — I don't have enough data yet to say anything reliable. I'll publish numbers once there are 30+ videos in the channel.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

How I built a pre-post QC gate that blocks Bluesky automation from self-revealing

MORINAGA — Sun, 28 Jun 2026 08:38:57 +0000

Three weeks into running a Bluesky queue from an automated content pipeline, I saw a post go out that referenced "the content pipeline" directly. Not egregiously — it was a passing phrase — but it was the kind of thing that reads differently on a social timeline than it does in a dev.to article. On dev.to, being honest about automation is a feature. On Bluesky, unprompted mentions of your own automation mechanism register as a red flag to human readers who are already primed to distrust content farms.

The JSONL-based queue I described earlier was working fine mechanically — entries generate, sit in the queue, and flush one at a time via a cron job. But there was no filter between the generation step and the post step. Whatever the prompt produced went into the queue, and whatever was at the front of the queue got posted. The Bluesky AT Protocol post API has no server-side content filter beyond spam detection, so the responsibility sits entirely with the client.

I spent a Saturday building bluesky-qc.mjs. It's a gate script that now runs as the first step in the posting workflow. Here's how it works.

The architecture: a gate between queue and post

Before bluesky-qc.mjs, the cron job was:

bluesky-post-queue.mjs → Bluesky API

After:

bluesky-qc.mjs → (PASS) bluesky-post-queue.mjs → Bluesky API

Both scripts read from the same content/bluesky-queue.jsonl file. The QC script walks entries in order, applies four gates to each, and either clears the first clean entry for the post script or moves failing entries to a rejection log. The post script then finds the first unposted entry in the queue — which, if QC just ran, should be a clean one.

In GitHub Actions this runs as pnpm bluesky:qc-then-post, a single composite command. If QC rejects everything in the queue and there's nothing clean to pass through, the workflow exits 0 without posting. Skipping a day is fine. Posting something that reads as automation-reveal isn't.

This connects to the same philosophy behind the Cloudflare Pages race fix I wrote about earlier — building explicit serialization into a pipeline that otherwise looks like it can run all steps in parallel.

Gate 1: vocabulary rejection

G1 is a compiled case-insensitive regex against a list of phrases that signal automated origin:

const REVEAL =
  /programmatic|content[\s-]pipeline|AI-curated|AI-generated|AI authorship|
   my sites|three (independent )?sites|pages generated|page count|records_in|
   llm_calls|build_duration|directory site|scaled content|batch test|
   120 pages|\bcron\b|autonomous agent|unsupervised PR|curat|generation|
   generate|disclos|thin pages|thin-content|\bprose\b|HCU|index vs|
   get dropped|pruning|bounce rate|citation rate|refresh signals|
   content experiments/ix

This is aggressive by design. "curat" catches "curated" and "curation" — both common in automated content contexts. "generate" catches anything mentioning generation. Posts occasionally trip G1 on phrases I'd consider acceptable, and I intentionally leave the regex tight. If a post needs to reference generation or curation to make its point, Bluesky probably isn't the right channel for that thought; dev.to is.

The pattern grew over time. I started with about 15 terms and added ~15 more after specific entries cleared an early version and still felt off when I reviewed the rejection log. The final regex is the output of four weeks of manually auditing what the gate should have caught.

One thing I added late: \bcron\b. Even the word "cron" — in a sentence like "I run this as a cron job" — is a signaling phrase for automation in a social context where the default assumption is that people are posting manually.

Gate 2: freshness, two parts

Staleness shows up in two distinct forms.

Stale phrasing: the entry uses time-relative language that was accurate when generated but isn't accurate when it finally posts.

const STALE =
  /\btoday\b|this week|yesterday|this morning|just\s+(announced|released|landed|launched|dropped)/i;

An entry might say "just dropped" about something that was new when the prompt ran but is three days old by post time. G2a catches this before it reaches the timeline.

Stale timestamp: the entry was created more than TTL_DAYS = 14 days ago. The queue sits ahead of a given entry if the entry was generated a while back and newer entries jumped ahead of it. Both created_at and generated_at field names are checked because the generation scripts in this repo aren't consistent:

const ts = entry.created_at || entry.generated_at || entry.createdAt;
if (ts) {
  const ageDays = (Date.now() - Date.parse(ts)) / 86400000;
  if (ageDays > TTL_DAYS) {
    reasons.push(
      `G2: ${Math.floor(ageDays)} days old (TTL ${TTL_DAYS})`
    );
  }
}

The 14-day TTL is based on my queue depth and posting rate. At one post per day, anything more than 14 entries deep in the queue was probably generated in a context that no longer applies — the tool being referenced may have had an update, the framing may feel dated.

Both sub-checks write separate rejection reasons so the log makes clear which flavor of staleness triggered.

Gate 3: engagement prediction (warn-only in v1)

Gate 3 uses data/bluesky-engagement-profile.json, generated weekly by bluesky-engagement-stats.mjs. That script pulls my last 300 posts from the Bluesky API, calculates a score per post as likes + 2×reposts + replies (reposts weighted higher as a stronger signal), and builds a breakdown by hashtag.

At post time, G3 looks at which hashtags appear in the pending entry and computes a predicted score relative to the baseline:

if (profile?.by_hashtag) {
  const tags = [...entry.text.matchAll(/#[A-Za-z][A-Za-z0-9_]+/g)]
    .map((m) => m[0].toLowerCase());
  const found = tags
    .map((t) => ({ t, m: profile.by_hashtag[t]?.median }))
    .filter((x) => x.m !== undefined);
  const baseline = profile.overall?.median ?? 0;
  if (found.length) {
    const predicted = Math.max(...found.map((x) => x.m));
    g3 = { predicted, baseline, tags: found.map((x) => x.t) };
  }
}

G3 is warn-only in v1. It logs the predicted vs. baseline score but doesn't reject. The reason: I don't have enough post history yet for the signal to be reliable. The shared Claude Haiku client I use for content generation has been running for about two months, and post volume is roughly one per day — that's around 60 data points. Median engagement is still low enough that G3's predictions are noisy.

When I flip G3 to hard-fail (the code has a comment marking where the threshold check goes), I expect it to catch G1/G2 survivors that technically pass the text checks but target hashtags that don't perform for this account. A comment in the code marks this:

// v1: warn-only. データ蓄積後に hard-fail へ昇格予定

The engagement stats script also breaks down by time window (30d, 60d, 90d), so the profile should become more useful as data accumulates over the next few months.

Gate 4: reserved for Codex

The fourth gate is unimplemented. The design intent is a --codex flag that pulls Codex for a quality pass on anything that cleared G1-G3 but still needs a final review. In the current setup, Codex runs at generation time through the article pipeline (the three-layer Codex protocol handles article quality) — but for Bluesky posts, the generation step doesn't include a Codex pass. G4 would close that gap.

I'm deferring G4 until G3 is stable, because adding latency and API cost to a cron that already runs three times a day doesn't make sense while the data foundations aren't solid yet.

What happens to rejected entries

Every failing entry gets appended to data/bluesky-qc-rejected.jsonl:

appendFileSync(
  REJECTED,
  JSON.stringify({
    ...entry,
    qc_reasons: reasons,
    qc_at: new Date().toISOString(),
  }) + "\n"
);

The rejected log serves two purposes: nothing is lost (entries can be salvaged by editing and re-adding to the queue), and it's the primary feedback mechanism for tuning generation prompts upstream. If G1 keeps hitting "content pipeline" in posts that were supposed to be tool roundups, the prompt that generates those posts is leaking pipeline jargon. That's a prompt fix, not a gate fix.

I review the rejected log about once a week, the same session where I run bluesky-engagement-stats.mjs to refresh the G3 profile. Together this takes about 20 minutes.

Differences from the post-deploy check pattern

The single CI pipeline this sits in already runs post-deploy checks after every Cloudflare Pages build. Those checks are about production correctness — did the right pages render, do the JSON-LD blocks validate, is the sitemap current. The Bluesky QC gate is about tone and context — does the text feel authentic for a social timeline.

The design principle is the same: gate as early as possible, make failures informative, never silently swallow errors. But the failure modes are different. A broken JSON-LD block is objectively wrong. A post that mentions "content pipeline" isn't wrong — it's contextually wrong for the audience.

What I'd do differently

Atomic state machine. The two-script design has a coordination gap: QC clears an entry, post script runs, if the network fails mid-post the entry stays at the front of the queue and QC re-evaluates it next run. This is harmless but wasteful. A single script that locks the entry state before attempting the post — and marks it as qc_passed in the JSONL — would eliminate the re-evaluation churn.

Classifier instead of regex. The REVEAL regex is 30+ terms and growing. The right long-term design is a small logistic regression or a few-shot Claude classifier trained on the rejected log once there are 100+ examples. I'll have enough data in about six months at current rejection rates. Until then, a regex is zero latency, zero cost, and deterministic — properties I'd be trading away for accuracy I don't yet need.

G3 as a hard gate sooner. The warn-only period for G3 feels too conservative in retrospect. I could have set a very permissive threshold (say, predicted score < 0.5 × baseline median) and still caught the clearly low-signal cases without needing extensive data. I'll revisit once I hit 90 days of post history.

FAQ

Why not filter at generation time rather than at post time?

I do — generation prompts include explicit instructions to avoid revealing automation. But prompts are probabilistic. The gate is deterministic. Both layers running in sequence is cheaper than relying on either alone.

What if the whole queue fails QC?

The script exits 0 with a log message. Nothing posts. The queue isn't cleared, so the next run tries again. If the queue stays empty for several consecutive days, the queue refill cron (bluesky-refill-queue.mjs) adds fresh entries.

How do you decide what goes into the REVEAL list?

Anything I'd feel awkward saying to a human who asked "did you write this?" — tool names for automated workflows, quantitative signals from CI pipelines, phrases that have no natural place in a first-person social post.

Does the gate ever false-positive on legitimate content?

Yes. "generate" catches legitimate uses of the word. Posts about generators, code generation tools, or electrical generation would trip G1. I accept that loss. Posts about those topics are edge cases for this particular Bluesky presence, and if they do come up I can edit the queue entry to rephrase before re-adding.

Related:

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

What I'm watching this week: OpenClaw, Trellis.2, Gemma 4, and two more

MORINAGA — Sat, 27 Jun 2026 22:11:27 +0000

Five open-source projects I pulled up and actually thought about this week. Not a comprehensive roundup — just what caught my attention as someone running three AI-curated directory sites on a $25/month budget.

OpenClaw: the most-starred repo in GitHub history

openclaw/openclaw just crossed 373,000 stars, making it the most-starred software project in GitHub history, overtaking React. It's a self-hosted AI assistant that routes to any model — local or API — and connects to over 50 messaging platforms: WhatsApp, Signal, iMessage, Slack, Discord, Telegram, and more, all from a single config file. Created by Peter Steinberger, it has 1,200+ contributors.

From my directory site angle, OpenClaw is exactly the kind of project that drives "Open Alternative To" traffic. People search for "self-hosted alternatives to [Claude API / ChatGPT]" and what they actually want is something like this — a local-first agent they control. The 300K+ star count without a polished marketing page tells me the latent demand is real. I'm going to add it to the OSS directory and watch whether the page indexes.

Claude Mythos Preview at 93.9% SWE-bench Verified

Anthropic published Claude Mythos Preview this week, which now leads the SWE-bench Verified leaderboard at 93.9%. For reference: Claude Opus 4.8 is at 88.6%, Claude Opus 4.7 Adaptive at 87.6%, and six months ago 80% was the ceiling. The trajectory is steep.

I'm running Claude Haiku 4.5 for my ETL and content generation — cost and latency matter more to me than ceiling performance. But I'm watching this number for a different reason: as coding agents approach 95%, the interesting question shifts from "can it write correct code?" to "can it understand architecture trade-offs without prompting?" SWE-bench doesn't measure that, and I don't know when that bar will move.

Gemma 4: Apache 2.0, agentic, multimodal, four sizes

Google released Gemma 4 in early April and made further announcements at Google I/O in May. The family: Effective 2B, Effective 4B, 26B MoE, and 31B Dense. All under Apache 2.0. The 31B model ranks #3 on the Arena AI open-model leaderboard; the 26B is #6.

What makes it relevant for my stack: native function-calling and structured JSON output are built in from day one, not retrofitted. The E2B model is designed for edge inference, which raises the question of whether it could replace Claude Haiku for my nightly ETL without a cost increase. The honest answer is I don't know — it depends on inference latency under load, and I haven't run that benchmark. I'll do it once I have a stable GPU budget to test against.

LLaDA2.0-Uni: discrete diffusion at 100B parameters

InclusionAI/LLaDA2.0-Uni (Ant Group) is a unified multimodal model — text generation, image understanding, and image generation — built on discrete diffusion instead of autoregressive decoding. The 100B flash variant uses MoE, making it the largest discrete diffusion model published so far.

The architecture is genuinely different from everything else: during training, tokens are randomly masked at varying rates; at inference you start from all-masked and run the reverse process. I'm not suggesting this is production-ready for most use cases today. But as a directional signal — "you don't have to do autoregressive decoding to get strong results at scale" — it's worth tracking. If this line of research matures over the next 12–18 months, it changes the inference cost curve in ways that matter for people running batch AI pipelines cheaply.

TRELLIS.2: image-to-3D in three seconds, MIT license

Microsoft released TRELLIS.2-4B on HuggingFace under MIT license. It converts a single image to a fully textured 3D asset: 512³ resolution in roughly 3 seconds, 1024³ in about 17 seconds, on an NVIDIA A100. Minimum 24GB VRAM.

I don't have an immediate use for 3D generation in my stack. But I'm watching because it fits the same pattern as the open-source video generation wave from late 2025: once the compute floor drops below "GPU you already own," the tool stops being a research artifact and becomes infrastructure. TRELLIS.2 isn't there yet at 24GB minimum, but the MIT license and HuggingFace availability suggest Microsoft is betting on community adoption rather than API gating. The next version will probably run on a 16GB card.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

5 things I noticed this week: CI cost, Bluesky QC, and CC0 licensing

MORINAGA — Sat, 27 Jun 2026 22:11:23 +0000

Five things I noticed or shipped this week while running the AI directory sites and a YouTube automation pipeline. No big announcements — mostly friction I hit and adjustments I made.

1. GitHub Actions was eating my free quota silently

I had a Bluesky posting cron running three times a day. Each run triggered a four-platform matrix build (three sites × multiple jobs). When I looked at my Actions minutes consumed this week, the math was embarrassing: 3 cron runs × 5-6 minutes each × 7 days = roughly 120 minutes/week just for posting a tweet-sized status update.

Two fixes. First, I changed the Bluesky cron from 0 */8 * * * to a single daily trigger — the queue buffers three posts regardless of how many times the cron fires, and posts don't go out faster than the queue feeds them. Second, I added a path filter so content-only commits (articles, copy edits) skip the four-way matrix build entirely. A new article doesn't need a full CI rebuild of all three Astro sites.

Actions quota is not infinite. Even on a paid plan, burning minutes on no-ops is a bad habit to get into before the repo scales.

2. Bluesky posts need a quality gate before they leave the queue

I added a QC gate to the Bluesky post pipeline this week — a step that reads each queued post, checks it against a short ruleset (no broken links, no expired announcements, no posts that reveal the automation stack in a tone that sounds like spam), and drops anything that fails before the cron fires.

The immediate trigger: I audited the outbox and found 17 posts that read like a bot talking to itself. Phrases like "🔁 queued" and "auto-generated" in a context where I had not disclosed that. Not illegal, but not the tone I want on a personal account.

The gate runs as a step before the actual bluesky post command. If it rejects a post, the item stays in the queue with a flagged status so I can review it manually. Net result: fewer posts per day (down from three to one or two), but ones I would not be embarrassed to have written manually.

3. Stopping model routing made the pipeline simpler

I wrote a YT script this week about removing model routing — the pattern where you send different content types to different AI models based on some classifier. I had been routing "short factual" queries to a faster/cheaper model and "synthesis" queries to a more capable one.

What I found after removing it: latency stayed basically the same, cost went up about 8%, and the code got significantly simpler. The routing classifier itself had edge cases. When the classifier misfired on a synthesis query and sent it to the cheaper model, the output was noticeably worse. The 8% cost increase to send everything to the capable model is cheaper than debugging routing bugs.

This is not a universal takeaway — at scale, routing probably pays off. At indie scale with a handful of daily API calls, the complexity cost is real and the savings are marginal.

4. Openverse CC0 filtering is not default — you have to opt in

I added image slides to the YouTube slide renderer this week using Openverse. The API returns results across multiple Creative Commons license types by default. For a monetized YouTube channel, using CC-BY images without visible on-screen attribution is a real licensing problem.

The filter I needed is license=cc0,pdm — not the default. Without it, you get CC-BY, CC-BY-SA, CC-BY-NC results mixed in with no indication they require credit. The API returns a license field per result, but if you're batch-processing slides and forget to filter upstream, you will miss one eventually.

A second issue: Openverse sometimes returns results pointing to images that have since been removed from the source host. The API returns 200 with metadata, but the actual image URL 404s. I added a requests.head() check before the slide renderer tries to download anything, and skip results that return non-200.

5. Self-hosted observability tools have a comfort vs. capability gap

I did a comparison of Netdata, SigNoz, and OpenObserve this week for the purpose of monitoring the three sites. All three install in under 10 minutes. The divergence shows up in what you're comfortable touching at 2am when something breaks.

Netdata is the most comfortable out of the box — it auto-discovers processes and starts charting immediately. SigNoz requires you to send OpenTelemetry traces explicitly, which means instrumenting your code first. OpenObserve is log-focused and works well if you're piping structured JSON logs, but its dashboard interface has a steeper learning curve than the other two.

For my current situation (Vercel + Cloudflare Pages, no VPS to instrument), all three are somewhat over-engineered. I ended up with a single Datadog free-tier integration for error alerting and leaving the self-hosted tools as a future option if the infrastructure changes.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Openverse CC0 images in a monetized YouTube pipeline: four things to check first

MORINAGA — Sat, 27 Jun 2026 22:10:51 +0000

I added an image slide type to the YouTube slide renderer for cases where a real photo contextualizes a slide better than a Mermaid diagram. The source is Openverse, the CC-licensed media search maintained by WordPress.org. Before it worked reliably in CI, I hit four issues that are not obvious from reading the Openverse API docs.

1. CC0 and PDM only — not CC-BY

Openverse returns results across multiple Creative Commons license types. The default search includes CC-BY, CC-BY-SA, CC-BY-NC, and others alongside CC0 and PDM. For a monetized YouTube channel, the practical split is:

CC0 and PDM: no attribution required. You can use these in commercial content without a credit line.
CC-BY and variants: attribution required. On a slide or thumbnail, that means visible on-screen credit — "Photo by X via Y" somewhere on the frame, readable on screen.

For slides in a fast-paced video, on-screen attribution text has to be legible but unobtrusive. CC-BY attribution on a dark-background slide is doable but adds layout complexity — the credit needs to clear the host overlay, stay below the heading, and still be readable at 1080p. More importantly, if you miss a single CC-BY image in a batch-generated video, you are distributing unlicensed content.

The filter I use is license=cc0,pdm — nothing else. Even though CC0 technically does not require attribution, I still add a courtesy credit caption on every image slide:

cap = f"{custom} · {attribution}" if custom else attribution  # always keep credit

The attribution string is formatted like "Title" · Creator · CC0 · Openverse. It appears at the bottom of the slide in muted text. Not legally required for CC0; still correct practice, especially for Openverse's long-term funding (the catalog depends on content creators seeing attribution from reuse).

2. The 200px minimum filter matters

The Openverse API can return results that include small thumbnails, low-resolution copies, and occasionally images that fail to load entirely. Without filtering, you get unusable frames in your videos.

im = Image.open(io.BytesIO(raw)).convert("RGB")
if min(im.size) < 200:   # skip tiny/thumbnail junk
    continue

The check is against the smaller dimension. An image that is 1800x180 pixels would fail this check — it's too narrow to fill a 1920x1080 slide usefully. 200px is a loose floor; in practice most results that pass it are at least 400x400. If the query term is obscure (narrow technical topics often return few results), you may exhaust all 12 results in a page and get a RuntimeError. The fallback in _visual_slide catches this and renders a heading-only card — the build continues.

page_size=12 is set explicitly because the Openverse default varies and results 0-11 give you enough candidates to find at least one usable image for common topics.

3. SSL certificate errors in CI

On Ubuntu GitHub Actions runners, plain ssl.create_default_context() sometimes fails HTTPS requests to Openverse with certificate verification errors. The reliable fix is certifi:

try:
    import certifi
    ctx = ssl.create_default_context(cafile=certifi.where())
except Exception:
    ctx = ssl.create_default_context()

The except branch keeps the code working locally where certifi may not be installed. In CI, certifi is pinned in the workflow's pip install step (pip install certifi Pillow), so the primary branch runs.

The same pattern applies to the image download request — both the API call and the image fetch use the same ctx. Without this, roughly one in ten CI runs fails on SSL at the image download step even if the API call succeeds, because the two requests can hit different CDN nodes.

4. Retry with backoff is necessary

The Openverse API has rate limits and occasional transient 5xx responses. A single request without retry logic fails silently in CI in a way that is hard to debug after the fact:

for attempt in range(3):
    try:
        with urllib.request.urlopen(req, timeout=timeout, context=ctx) as r:
            data = json.load(r)
        break
    except Exception:
        if attempt == 2:
            raise
        time.sleep(1.5 * (attempt + 1))

Three attempts with 1.5s and 3.0s waits covers most transient failures. The timeout on urlopen is 15 seconds — enough for a slow CDN response, not so long that it blocks the entire video build if Openverse is down.

The image download loop (iterating through results) has no retry per-image, because if one image URL fails you just move to the next candidate. The retry logic sits at the API query level, where the cost of a transient failure is losing all candidates at once.

What the complete integration looks like

In the slide spec JSON, an image slide looks like:

{
  "kind": "image",
  "heading": "Self-hosted observability stack",
  "query": "server rack data center monitoring",
  "caption": "Grafana + Prometheus in production"
}

The query drives the Openverse search. The caption prefixes the attribution string. When the fetch succeeds, the slide shows the heading, the photo centered in the content area, and the attribution credit at the bottom. When it fails — query too specific, Openverse down, all results too small — the slide shows the heading alone. Either way the video builds.

The design principle is the same one behind the three-tier content quality ladder for the directory ETL: the build never blocks on an enrichment step. Degraded output is better than a failed pipeline.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

What I learned adding diagram and chart slides to a CI-rendered YouTube pipeline

MORINAGA — Sat, 27 Jun 2026 08:24:01 +0000

The conclusion first: pre-rendering diagrams and charts to PNG before compositing them onto slides — rather than generating visual content inline or inside ffmpeg — is the right architecture for a CI video pipeline. The tooling gap between Chromium-backed Mermaid rendering, headless matplotlib, and ffmpeg's static frame expectation makes a shared PNG handoff the only approach that keeps each piece testable and replaceable.

I added three new slide types to the YouTube slide renderer last week: diagram (Mermaid flowcharts and sequence diagrams), chart (branded horizontal bar charts via matplotlib), and image (license-clear photos from Openverse). The existing slides — title, bullets, table, tool, outro — all draw directly with Pillow. These three render externally, produce a PNG, and get pasted into the same Pillow canvas. Same output contract, different render path.

Why pre-render instead of embed

The two-host pipeline assembles video by compositing a still image for each dialogue segment, synthesizing audio with edge-tts, and using ffmpeg to concatenate the clips. ffmpeg expects the still to be a file or a stream of identical frames — it does not run JavaScript, and it cannot call a browser mid-concat.

Mermaid runs through Puppeteer and Chromium. Pillow draws directly on a numpy-backed image. There is no in-process way to make these talk. The only clean option is: mmdc produces a PNG, Pillow pastes the PNG.

matplotlib is different — it could theoretically produce an image buffer in the same process. But having a consistent "render to PNG file, paste PNG file" pattern for all three visual types means they share the same _visual_slide scaffold and graceful-degradation path. One code path is better than two.

Mermaid via mmdc: the CI-specific configuration

render_mermaid() in visuals.py writes two config files before calling mmdc:

cfg = os.path.join(tempfile.gettempdir(), "bs-mmd-theme.json")
with open(cfg, "w", encoding="utf-8") as f:
    json.dump(_MMD_THEME, f)

pcfg = os.path.join(tempfile.gettempdir(), "bs-puppeteer.json")
with open(pcfg, "w", encoding="utf-8") as f:
    json.dump({"args": ["--no-sandbox", "--disable-setuid-sandbox"]}, f)

The puppeteer config is the one that bit me first. Chromium refuses to start as root without --no-sandbox, and GitHub Actions runs as root inside the Ubuntu container. Without --disable-setuid-sandbox, it will also fail on containers where setuid is restricted. Both flags are needed.

The theme config uses Mermaid's base theme, not dark or forest. The other named themes override themeVariables, so color injection does not work reliably with them. Only base respects the custom palette (confirmed in the Mermaid.js theme docs):

_MMD_THEME = {
    "theme": "base",
    "themeVariables": {
        "primaryColor": PANEL, "primaryTextColor": INK, "primaryBorderColor": ACCENT,
        "lineColor": MUTED, "secondaryColor": "#1B2B4A", "tertiaryColor": BG,
        "fontFamily": "Arial", "fontSize": "22px",
        "clusterBkg": PANEL, "clusterBorder": ACCENT,
    },
}

The mmdc binary resolution tries three paths:

def _mmdc_cmd() -> list[str]:
    local = os.path.join(_DIR, "node_modules", ".bin", "mmdc")
    if os.path.isfile(local):
        return [local]
    if shutil.which("mmdc"):
        return ["mmdc"]
    return ["npx", "--yes", "@mermaid-js/mermaid-cli"]

The npx fallback is correct for CI: the GitHub Actions workflow installs @mermaid-js/mermaid-cli as a dev dependency, so local node_modules is the hot path. The npx branch exists for local dev where you have not run npm install. Do not make npx the primary path — it downloads the package on every invocation, which adds 20-30 seconds per diagram slide in a cold runner.

The -w 1600 width flag matters. At 1920x1080, the content area after chrome and heading is roughly 1700x700. Rendering at 1600px wide gives mmdc enough resolution to produce readable text without scaling artifacts when _paste_visual() thumbnails it into the slot.

matplotlib horizontal bars with a custom dark palette

render_chart() takes a simple spec:

# spec = {"items": [["Tool A", 41200], ["Tool B", 28900], ...], "unit": "stars", "highlight": 0}

Three things that tripped me up when trying to match the slide background:

First: matplotlib.use("Agg") must be called before import matplotlib.pyplot as plt. In Python, the backend selection call and the pyplot import are order-dependent — if you call use("Agg") after pyplot is imported, it either silently fails or raises. The function imports matplotlib inside the function body to avoid this at module load time:

def render_chart(spec, out_png):
    import matplotlib
    matplotlib.use("Agg")
    import matplotlib.pyplot as plt

Second: setting the background requires two separate calls:

fig.patch.set_facecolor(BG)
ax.set_facecolor(BG)

fig.patch is the outer figure background; ax is the axes area. Missing either one leaves a white rectangle where the background should be dark navy.

Third: plt.tight_layout(pad=1.2) is not enough on its own. Adding bbox_inches="tight" to fig.savefig() is required to clip the white padding matplotlib adds around the bounding box by default. Without it, the saved PNG has a white border that composites badly onto the dark slide background.

The highlight index accentuates one bar in ACCENT2 (green) instead of ACCENT (blue). The spec author sets it to mark the bar that is the point of the slide — the tool with the most stars, or the benchmark winner. It is optional; when absent, all bars render in blue.

The `_visual_slide` scaffold and graceful degradation

All three types share the same scaffold:

def _visual_slide(spec, render_fn):
    img, d = _base()
    _chrome(img, d, spec.get("page", ""))
    heading = spec.get("heading", "")
    if heading:
        d.text((MARGIN, 150), heading, font=font(56, "bold"), fill=INK)
        d.rectangle([MARGIN, 228, MARGIN + 120, 236], fill=ACCENT)
    import tempfile
    fd, tmp = tempfile.mkstemp(suffix=".png")
    os.close(fd)
    try:
        render_fn(tmp)
        _paste_visual(img, tmp, top=270 if heading else 150)
    except Exception as e:
        sys.stderr.write(f"WARN: visual render failed ({e})\n")
        d.text((MARGIN, 420), "[visual unavailable]", font=font(40), fill=MUTED)
    finally:
        if os.path.exists(tmp):
            os.unlink(tmp)
    return img

The tempfile.mkstemp pattern (create and close descriptor separately) is deliberately cross-platform: on Windows, NamedTemporaryFile with delete=False sometimes holds a lock that prevents a subprocess from writing to the same path. mkstemp avoids this. On Linux it makes no practical difference, but the code runs locally on macOS too.

The except Exception fallback is intentional. If mmdc is not installed, if matplotlib is not present, or if Openverse returns no usable images, the slide renders as a clean heading card with a muted "[visual unavailable]" placeholder. The video build continues. A missing diagram is not a build failure.

This matches the design of the rest of the pipeline: the CI job should produce a video even when external dependencies are partially absent. A video with a placeholder slide is reviewable; a failed build is not.

Where the approach has limits

mmdc startup cost. Each mmdc call launches a Chromium process, renders the SVG, and exits. That takes 2-4 seconds per diagram on the GitHub Actions ubuntu-latest runner. A video spec with five diagram slides adds 10-20 seconds to the build. For the current video lengths (15-25 segments), this is acceptable. If the format grew to 50+ slides, pre-rendering all diagrams in parallel before the main loop would matter.

matplotlib version pinning. The chart code relies on a specific call signature for barh() and tight_layout(). matplotlib has changed these interfaces across minor versions. The workflow pins matplotlib==3.9.* to avoid surprises in future runner image updates.

Mermaid syntax drift. mmdc's supported Mermaid syntax depends on the installed package version. Sequence diagrams work. gitGraph works. More recent additions (xychart-beta, sankey-beta) require a newer mmdc version than what ships with npm without explicit pinning. The solution is to add @mermaid-js/mermaid-cli@^11 to the workflow's npm install step rather than relying on the default version.

No preview in local dev. The analytics feedback loop tells me which slides hold attention; I cannot see a diagram slide without building the full video. Adding a --demo flag to slides.py that renders a single slide type from a spec file would help iterate faster, but I have not built it yet. For now the iteration loop is: edit the JSON spec, push a commit, wait for the CI video build, scrub to the diagram segment. That is slow. The single CI pipeline takes about 4 minutes end-to-end for a 20-segment video, which is reasonable for final validation but too slow for visual design iteration. A local --demo render that skips TTS and ffmpeg and just opens the PNG would cut that to under 10 seconds.

FAQ

Does mmdc need Node.js pre-installed on the runner?
Yes. The ubuntu-latest runner ships with Node.js 20, so the workflow just needs npm install -D @mermaid-js/mermaid-cli in a setup step.

Can I use Mermaid's dark theme instead of base with custom variables?
No. The dark theme resets most themeVariables to its own palette. The only theme that lets you fully override colors via themeVariables is base.

What happens if Openverse returns zero results for a query?
fetch_image() raises RuntimeError("no usable Openverse image for query: ..."). The _visual_slide scaffold catches it and renders a heading-only fallback card. No build failure.

Why horizontal bar charts instead of vertical?
Longer tool or model names truncate or require rotation on vertical axes. Horizontal bars handle 20-40 character labels without layout code.

Is the PNG-to-Pillow paste lossless?
Pillow opens the PNG fully decoded at 24-bit color depth. The save to final PNG at the end of slide composition uses default Pillow compression (level 6), which is lossless. There is no quality loss from the round trip.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Netdata vs SigNoz vs OpenObserve: self-hosted observability for indie projects

MORINAGA — Sat, 27 Jun 2026 08:23:57 +0000

I've been building out the ossfind.com OSS alternatives directory, and observability tools come up constantly as a category where the SaaS-to-OSS migration question is genuinely interesting. Datadog is the canonical expensive monitoring platform; replacing it with open source isn't a simple one-for-one swap. I spent time researching three projects that together — or individually — cover a meaningful slice of what Datadog does.

These aren't toy tools. All three have serious GitHub star counts and production deployments. The question for an indie project isn't whether they're capable enough, it's whether they're operable enough given a small team and a tight budget.

Netdata: fastest time to value on this list

GitHub stars: ~79k. License: GPL-3.0.

Netdata installs with one command and immediately starts collecting ~800 pre-built metrics across your host, containers, and services. No config file required to get a working dashboard. That's a legitimately different experience from standing up Prometheus from scratch.

GPL-3.0 is clean for self-hosting. The copyleft only matters if you build a commercial product on top of Netdata and sell it. Running it internally doesn't trigger the requirement. The scope limit to understand: Netdata is infrastructure metrics and alerting. If distributed traces or centralized log search are the Datadog features you actually depend on, Netdata alone doesn't close that gap.

For an indie project running a few Vercel deployments and a VPS, Netdata on the VPS handles host-level observability without any setup cost. That's its actual sweet spot.

SigNoz: the all-in-one APM stack

GitHub stars: ~27k. License: see LICENSE (dual — a non-AGPL community edition and a paid enterprise tier).

SigNoz is the most ambitious of the three. It bundles infrastructure metrics, distributed traces (via OpenTelemetry), and log management in a single product with a single login. That's the Datadog feature set as a self-hosted stack.

The OpenTelemetry integration is worth noting: your instrumentation code sends standard OTLP data, and you could swap SigNoz for any compatible backend later. That's a better position than adopting a proprietary agent format.

The tradeoff is operational weight. SigNoz runs multiple containers — ClickHouse, query service, frontend. A basic deployment consumes meaningful memory. That's fine for a team with dedicated infrastructure, but for a solo project where the "server" is a $6 Hetzner box also running a database and three Astro sites, it's a lot to manage.

The license situation needs attention before any commercial deployment. The community edition is open-source. The enterprise tier is not. Check the LICENSE file rather than assuming one way or the other.

OpenObserve: the log-first approach

GitHub stars: ~19k. License: AGPL-3.0.

OpenObserve optimizes for storage efficiency on logs. The project claims significant storage savings compared to Elasticsearch-based setups for log data — that's its differentiator. It also supports traces and metrics, but log aggregation is where it's strongest.

AGPL-3.0 means the same copyleft as SigNoz's community license: use it internally without restriction, but if you expose it as a SaaS product, you need to open-source that service. For a directory site or a personal project, that's a non-issue.

The use case I'd reach for it: a project where log volume is high and storage cost is a real constraint. If you're paying for a logging SaaS and the bill is mostly driven by log data volume, OpenObserve is where I'd look first.

How I'd pick between them

Need	Tool
Host metrics, minimal ops	Netdata
Full APM (metrics + traces + logs)	SigNoz
High log volume, storage efficiency	OpenObserve
Budget under $10/month server	Netdata only

For most indie projects in 2026, the honest answer is that full observability — metrics, traces, and logs — isn't the first priority. Getting something running cheaply is. Netdata solves the "is my server on fire" question immediately; that's usually enough to start.

The moment distributed tracing becomes useful is when you have multiple services talking to each other and you can't tell which one is slow. A single Astro site served from Vercel edge functions with a Turso database doesn't really need distributed tracing. A microservices setup does.

What I actually run

For my three sites, I use none of these currently. The sites are static Astro deployments on Vercel; Vercel's built-in analytics and the GitHub Actions run logs cover 90% of what I need right now. I'll revisit when I have a persistent service that generates actual log volume worth searching.

The research fed directly into the ossfind.com Datadog alternatives page, which lists these three plus Grafana and HyperDX with more detail on each.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

How I built the OSS alternatives directory: GitHub ETL, Turso, and the UPSERT trap I hit

MORINAGA — Fri, 26 Jun 2026 22:12:58 +0000

When I launched three programmatic directory sites in April 2026, the open-source alternatives site had the most interesting data model. The AI tools directory indexes HuggingFace models — that's a pull from one API. The indie games directory reads Steam. But the OSS alternatives site has to answer a different question: for this SaaS product, which open-source repos actually cover the same use case, and how do they compare?

Getting that right required a two-phase ETL approach, a careful UPSERT strategy I initially got wrong, and some deliberate choices about where to use Claude Haiku and where to use a fallback template.

What the data model looks like

Three tables in Turso libSQL:

saas — the SaaS tool being replaced (Datadog, Notion, Figma, etc.)
alternatives — GitHub repos that serve the same use case, linked by saas_slug
saas_content — Claude-generated per-entry text: an intro, comparison notes, and migration tips

The alternatives table stores everything the GitHub API returns that matters for a directory: stars, forks, language, license, last_pushed, description. The saas_content table stores only what Claude adds — the editorial layer that turns raw repo metadata into something useful.

The full export lives in a JSON file that Astro reads at build time. No database connection at build. The ETL pipeline and the Astro build are separate processes.

Phase 1: seeding from JSON

The first time the site runs on a new machine, there's no database. Rather than block a local build on a live GitHub API pass, I wrote a seed.ts script that bootstraps the database from a hand-curated saas.json file.

The JSON contains: SaaS name, slug, homepage, category, and a list of owner/repo strings. Stars, forks, license, and last_pushed are deliberately omitted — they'll come from the live fetch. What I do include in JSON is pre-polished content for some entries where the Claude default output was weak.

for (const e of entries) {
  await db.execute({
    sql: `INSERT INTO saas (slug, name, homepage, category, fetched_at)
          VALUES (?, ?, ?, ?, ?)
          ON CONFLICT(slug) DO NOTHING`,
    args: [e.slug, e.name, e.homepage, e.category, now],
  });

  for (const a of e.alternatives) {
    await db.execute({
      sql: `INSERT INTO alternatives (saas_slug, repo, name, description, ...)
            VALUES (?, ?, ?, ?, ...)
            ON CONFLICT(saas_slug, repo) DO NOTHING`,
      args: [e.slug, a.repo, a.name, a.description, ...],
    });
  }
}

DO NOTHING on conflict for alternatives is correct: once GitHub data is live, the seed shouldn't clobber fresh stars counts with the static values from the JSON. But for saas_content, I initially used the same DO NOTHING — and that was a mistake I'll get to below.

Phase 2: live GitHub data

fetch-alternatives.ts calls the GitHub REST API for every owner/repo in the database and upserts the live fields. Unlike the seed, this is DO UPDATE — we want fresh data.

The sleep interval is 100ms between GitHub API calls. For an authenticated token that rate limit is conservative (GitHub's REST API allows 5000 requests per hour for authenticated users, so 100ms is well under the minimum gap needed). Unauthenticated would be 60 per hour, which is 60 seconds per call — completely impractical at scale. The monorepo authenticates with a secret in GitHub Actions.

Errors per-repo are caught and logged but don't abort the batch:

for (const repoFull of s.alternatives) {
  const [owner, name] = repoFull.split("/");
  try {
    const r = await getRepo(owner, name);
    await db.execute({
      sql: `INSERT INTO alternatives (saas_slug, repo, name, description, stars,
              forks, language, license, last_pushed, url, fetched_at)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
            ON CONFLICT(saas_slug, repo) DO UPDATE SET
              description = excluded.description,
              stars = excluded.stars,
              forks = excluded.forks,
              language = excluded.language,
              license = excluded.license,
              last_pushed = excluded.last_pushed,
              fetched_at = excluded.fetched_at`,
      args: [
        s.slug, repoFull, r.name, r.description,
        r.stargazers_count, r.forks_count,
        r.language, r.license?.spdx_id ?? null,
        r.pushed_at, r.html_url, now,
      ],
    });
    await sleep(100);
  } catch (err) {
    console.error(`  ! Failed ${repoFull}:`, err instanceof Error ? err.message : err);
  }
}

One field worth noting: r.license?.spdx_id returns null when GitHub sees a license file but can't identify the SPDX identifier. That happens more than you'd expect with non-standard licenses. I render those rows with "see repo" instead of a badge so I'm not misleading visitors about the license type.

Content generation with Claude Haiku

After the GitHub data is fresh, generate-content.ts queries for SaaS entries that either have no content row or whose model_used column is 'fallback-template' or 'seeded-from-json'. For each, it asks Claude Haiku for:

intro — 2 sentences on what the SaaS is and why teams seek OSS alternatives
comparison_notes — 2-3 sentences on actual tradeoffs (self-hosting overhead, feature gaps)
migration_tips — a 2-4 item array of concrete migration steps

I use the shared Claude Haiku client with system-prompt caching here. The system prompt is identical for every call in a batch, so caching it saves input tokens on all subsequent calls. On a 50-entry pass, the cost difference is real.

The fallback template — which runs when ANTHROPIC_API_KEY is absent — generates deterministic placeholder text. This matters for CI: the Astro build needs a content row for every SaaS entry. Missing content produces a blank page, which would then trigger the noindex gate I use for thin programmatic pages.

The three-tier content quality ladder I described earlier puts these generated entries at the middle tier — better than the raw repo description, worse than hand-edited content.

The UPSERT trap

Original seed.ts for saas_content:

INSERT INTO saas_content (saas_slug, intro, comparison_notes, migration_tips, generated_at, model_used)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(saas_slug) DO NOTHING

That looked safe. But the problem was subtle. When I seeded with model_used = null (the original JSON had no field), generate-content.ts queried:

SELECT slug FROM saas s
LEFT JOIN saas_content c ON c.saas_slug = s.slug
WHERE c.saas_slug IS NULL
   OR c.model_used IN ('fallback-template', 'seeded-from-json')

Rows seeded with model_used = null didn't match either condition. They also weren't NULL (the row existed). So they got skipped by the generator — but the seed DO NOTHING also prevented the polished JSON content from landing, because a fallback-template row had already been written by an earlier run.

The fix was two parts:

Seed.ts now uses DO UPDATE for saas_content, not DO NOTHING. Polished JSON content always wins.
The JSON schema requires model_used to be set explicitly — 'seeded-from-json' for automatic entries, 'claude-routine-polish' for hand-checked ones. The generator's WHERE clause excludes both.

ON CONFLICT(saas_slug) DO UPDATE SET
  intro = excluded.intro,
  comparison_notes = excluded.comparison_notes,
  migration_tips = excluded.migration_tips,
  generated_at = excluded.generated_at,
  model_used = excluded.model_used

This pattern — using model_used as a status field to coordinate between ETL phases — also showed up in the AI tools directory's fallback entry upgrade work. The lesson there was the same: never let an ETL pass silently skip a row because the status field was written inconsistently.

The Astro page structure

Each SaaS entry renders as a static page at /alternatives/[saas]/. The renderer reads from saas.json, assembles a grid of alternatives sorted by stars, and inlines the Claude-generated comparison notes. Each entry shows a license badge, language indicator, and last_pushed date formatted as a relative time string.

The grid intentionally doesn't paginate at the SaaS level. I capped entries per SaaS at 8. More than that becomes noise — the directory's value is curation, not exhaustiveness. The E-E-A-T transparency pages include a methodology note explaining what that cap means for each category.

What I'd change

Store raw GitHub JSON alongside derived columns. Currently each ETL adds derived fields: stars, forks, license, last_pushed. When I later wanted a "has_recent_releases" signal, I had to add a full new API call. If I'd kept the raw response in a JSONB/TEXT column, json_extract(raw, '$.has_wiki') would have been enough.

Add a deprecated_at field. When a repo gets deleted or renamed, the ETL call returns a 404 and the code just logs it. The row stays in the database with increasingly stale data. A deprecated_at timestamp would let the page renderer show a warning and let the content team decide whether to replace or remove the entry.

Parallelize generate-content with a rate-limit counter. The current sequential loop takes a noticeable number of minutes on a cold run with 100+ entries. Batching 10 concurrent Haiku calls with a shared counter that throttles at the API limit would be 5-10x faster without touching cost.

FAQ

Why Turso instead of a hosted Postgres?
Turso's edge replicas are in the same regions as Vercel's serverless functions, so read latency is low. The cost for my usage tier is also lower than a comparable Postgres instance. The full comparison is here.

Do you need a paid GitHub plan to avoid rate limits?
No. A free personal access token gives 5000 requests per hour — enough to fetch metadata for several hundred repos in a single daily cron run. The 60/hr unauthenticated limit would not work at any meaningful scale.

How do you prevent Claude costs from escalating?
System-prompt caching amortises the per-call cost across the batch. I also set max_tokens: 1024 for each call, which caps output length. The biggest lever is the model_used status field: entries that already have good content don't get regenerated.

What happens if a GitHub repo is deleted?
Right now the row goes stale silently. The fetch fails, the error is logged, and the next build still renders the row with whatever data the last successful fetch stored. Adding a 404-specific handler that sets deprecated_at is on the backlog.

DEV Community: MORINAGA

Four filters I apply when pulling HuggingFace models into an AI tools directory

Filter 1: pipeline tag — only end-user-facing tasks

Filter 2: minimum likes threshold — filtering out abandoned and test uploads

Filter 3: last-modified recency — flagging dormant models

Filter 4: gated and private models — inclusion requires public weights

What the chain still doesn't solve

Why I'm betting on Claude Code over Cursor for a solo dev pipeline

The bet, stated plainly

What pushed me toward Claude Code

What I'm giving up

The counterargument I take seriously

The cost structure

How I partition the workflow in practice

What would change my mind

The December 2026 checkpoint

FAQ

How I verify affiliate CTAs are actually rendering in production

What the script does

The sitemap crawl

Checking ads.txt and affiliate strings

Why this is better than a visual check

What I'd add

How I built a content quality gate that stops bad articles before they publish

Why a lint gate and not a manual review

What the gate checks

Strict versus lenient mode

Title duplicate detection across the repo

What actually got caught

What I'd do differently

FAQ

Two-host AI dialogue specs: how I structure YouTube longform scripts with A/B speaker JSON

What the spec looks like

How the build works

What the spec generator produces

What I haven't solved yet

How I built a pre-post QC gate that blocks Bluesky automation from self-revealing

The architecture: a gate between queue and post

Gate 1: vocabulary rejection

Gate 2: freshness, two parts

Gate 3: engagement prediction (warn-only in v1)

Gate 4: reserved for Codex

What happens to rejected entries

Differences from the post-deploy check pattern

What I'd do differently

FAQ

What I'm watching this week: OpenClaw, Trellis.2, Gemma 4, and two more

OpenClaw: the most-starred repo in GitHub history

Claude Mythos Preview at 93.9% SWE-bench Verified

Gemma 4: Apache 2.0, agentic, multimodal, four sizes

LLaDA2.0-Uni: discrete diffusion at 100B parameters

TRELLIS.2: image-to-3D in three seconds, MIT license

5 things I noticed this week: CI cost, Bluesky QC, and CC0 licensing

1. GitHub Actions was eating my free quota silently

2. Bluesky posts need a quality gate before they leave the queue

3. Stopping model routing made the pipeline simpler

4. Openverse CC0 filtering is not default — you have to opt in

5. Self-hosted observability tools have a comfort vs. capability gap

Openverse CC0 images in a monetized YouTube pipeline: four things to check first

1. CC0 and PDM only — not CC-BY

2. The 200px minimum filter matters

3. SSL certificate errors in CI

4. Retry with backoff is necessary

What the complete integration looks like

What I learned adding diagram and chart slides to a CI-rendered YouTube pipeline

Why pre-render instead of embed

Mermaid via mmdc: the CI-specific configuration

matplotlib horizontal bars with a custom dark palette

The _visual_slide scaffold and graceful degradation

Where the approach has limits

FAQ

Netdata vs SigNoz vs OpenObserve: self-hosted observability for indie projects

Netdata: fastest time to value on this list

SigNoz: the all-in-one APM stack

OpenObserve: the log-first approach

How I'd pick between them

What I actually run

How I built the OSS alternatives directory: GitHub ETL, Turso, and the UPSERT trap I hit

What the data model looks like

Phase 1: seeding from JSON

The `_visual_slide` scaffold and graceful degradation