DEV Community: MORINAGA

How I kept 62 of 80 programmatic pages alive while hiding them from Google

MORINAGA — Wed, 10 Jun 2026 22:17:34 +0000

After my second AdSense rejection for scaled content, I had two options for the thin pages on Open Alternative To: delete them and accept 404s on any inbound links, or keep them alive while hiding them from Google's quality evaluation. I chose the second.

The reasoning: I have links pointing at some of these URLs — from earlier articles in this series, from social posts, from internal site navigation. A 404 would break all of them. The pages aren't wrong, they're just thin. The correct signal to Google is "don't evaluate these" rather than "these don't exist."

The isCurated gate

The gate lives in apps/oss-alternatives/src/lib/curation.ts:

export const CURATION = {
  MIN_ALTERNATIVES: 4,
  MIN_TOP_STARS: 1000,
  MIN_INTRO_LEN: 80,
} as const;

export function isCurated(s: SaasEntry): boolean {
  if (!s.intro || s.intro.length < CURATION.MIN_INTRO_LEN) return false;
  const alts = s.alternatives ?? [];
  if (alts.length < CURATION.MIN_ALTERNATIVES) return false;
  const topStars = alts.reduce((m, a) => Math.max(m, a.stars ?? 0), 0);
  if (topStars < CURATION.MIN_TOP_STARS) return false;
  return true;
}

Three conditions, all required:

At least 4 open-source alternatives listed — a comparison page with fewer entries is barely a comparison
Top alternative has 1,000+ GitHub stars — filters out obscure or unmaintained projects that don't demonstrate the category's depth
Intro text is at least 80 characters — rules out the fallback-template content that the ETL quality ladder writes when Claude is unavailable

These are objective thresholds, not hand-picked entries. The gate runs automatically at every Astro build. Entries that gain another alternative or get a longer intro in the next ETL run will silently cross the threshold and become discoverable without any manual action.

Currently: 18 of 80 entries pass. That's the real data state, not a target. The nightly ETL upgrades entries progressively; the curated count will grow as the content improves.

Why the gate lives in its own module

saas.ts — where the main data access code lives — imports @libsql/client to query Turso. Any module that imports saas.ts at the value level picks up that dependency. Astro's static page bundles can't include server-only DB dependencies, so they'd fail to build.

The solution: curation.ts imports only types from saas.ts:

import type { SaasEntry } from "./saas.ts";

TypeScript erases type imports at compile time. At runtime, curation.ts has no external dependencies — it's a pure computation module that Astro can safely include in static page bundles. saas.ts stays server-side-only, imported only in getStaticPaths where the DB dependency is expected.

This split-by-dependency-type pattern comes up regularly in Astro monorepos. Anything that touches a runtime external goes server-side; the pure logic you need in both places gets its own module.

Four discovery surfaces gated on the same function

A page being "hidden" means four things happen simultaneously:

1. noindex meta tag — Base.astro checks isCurated(entry) and adds <meta name="robots" content="noindex, nofollow"> for entries that don't pass.

2. Sitemap exclusion — astro.config.mjs has a sitemap filter applying the same threshold logic. This is the one awkward part: astro.config.mjs can't import from src/, so the threshold values are duplicated. I put // KEEP IN SYNC: curation.ts on both. Changing the thresholds in one place without updating the other would produce a sitemap that disagrees with the noindex tags — some pages would be submitted to Google while simultaneously declaring noindex.

3. RSS feed — the feed only includes curated entries. Non-curated pages won't surface in feed readers as new content.

4. Internal navigation — homepage category cards, footer category links, breadcrumb paths, and "related alternatives" widgets all filter through isCurated. A direct link from outside the site still reaches the page. But browsing the site organically won't surface non-curated entries.

The category layer

Categories follow the same logic. A category is only indexable if it has at least two curated entries (CATEGORY_MIN_CURATED = 2). Categories below that threshold still generate pages — preserving any external links to category URLs — but they're noindex and excluded from the sitemap, homepage, and footer navigation.

Right now, only one category (customer-support) meets the threshold. That's the honest state of the data: the site has broad coverage but thin editorial depth across most categories. As the ETL runs and more entries cross the curation threshold, more categories will become indexable automatically.

What changes automatically

The gate is deterministic and evaluated at build time from live DB data. When foss-alternative-to-figma gains its fourth alternative and Claude Haiku generates a 90-character intro in the next nightly run, the following Astro build will automatically include it in the sitemap, remove its noindex tag, and add it to the relevant category card and footer link.

The only thing that doesn't update automatically is the duplicate threshold in astro.config.mjs. I'll eventually extract the constants to a shared JSON file that both curation.ts and astro.config.mjs read, eliminating the sync risk. For now the comment is the guard.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Why I'm abandoning AdSense on two sites and betting on affiliate monetization

MORINAGA — Wed, 10 Jun 2026 22:16:50 +0000

The first AdSense rejection was predictable. I'd launched three directory sites on Vercel and hadn't added custom domains immediately. Google won't approve a *.vercel.app site — the subdomain pattern can't carry a credible publisher identity and the policy requirement for a real contact address on the privacy page can't be met on a free subdomain.

Custom domains fixed that. I resubmitted.

Two weeks later: rejected again. This time for "valuable inventory," which is AdSense's way of saying the content doesn't meet the quality bar they need to place ads against. The reviewer flagged scaled content. Open Alternative To has 80 pages for 80 different paid tools. Even though Claude Haiku generates genuine editorial text for each one, the programmatic pattern triggered AdSense's classifier.

That second rejection forced me to actually run the economics I'd been deferring.

The asymmetry between affiliate and AdSense for a zero-traffic site

AdSense has an approval gate. Affiliate programs don't.

For a site in month one, that asymmetry is the entire decision. Display ad revenue on a brand-new site with essentially no traffic is effectively zero regardless of whether you're approved — there's nothing to monetize. The path to positive earnings requires: getting approved, building traffic, then earning CPM-based revenue at scale.

Affiliate revenue has no approval step. The first conversion earns commission the day the link is live. The earning curve is still terrible at low traffic, but the timeline starts earlier.

I've been deliberately honest in this series about not having numbers to report yet. The sites launched April 23, 2026; I'll publish month-one metrics in June. But the structural argument for pivoting now — before I have revenue data — is that the two monetization models have different minimum viable conditions. AdSense requires approval. Affiliate requires a user who clicks and buys.

Why I didn't pivot all three sites

Three sites, three different audiences:

Site	Primary intent	Monetization strategy	Rationale
Top AI Tools	Discover and adopt AI tools	Affiliate (Amazon, SaaS programs)	Purchase intent — evaluating paid tools
Find Games Like	Find similar indie games	Affiliate (Steam, Humble Bundle)	Purchase intent — close to a buy decision
Open Alternative To	Replace paid software with open-source	AdSense (when approved)	Anti-purchase intent — display ads monetize page views

The pivot logic turns on purchase intent. Someone browsing AI tools is probably evaluating whether to pay for a Pro plan. Someone looking for games similar to one they liked is close to a Steam purchase. Affiliate commissions trigger on exactly those decisions — the user was already considering the purchase.

The OSS alternatives audience is explicitly trying to not spend money. An affiliate link for "buy the paid version you were trying to avoid" is a misalignment. Display ads monetize the page view regardless of purchase intent, so AdSense is the structurally correct model for Open Alternative To — when the editorial quality clears approval.

This means ossfind stays on the quality-improvement track. I'm implementing a content quality gate that limits which pages are indexable, reducing the scaled-content signal that triggered the rejection. The target: resubmit with a smaller set of genuinely thick pages and the rest marked noindex.

The implementation: monetization mode as env var, not deletion

The cleanest part of this pivot was choosing not to delete the AdSense components. Deletion would make the decision permanent before I have revenue data. Instead I added a PUBLIC_MONETIZATION_MODE env var to the shared monetization package:

export type MonetizationMode = "adsense" | "affiliate";

export function getMonetization(): MonetizationConfig {
  const mode: MonetizationMode =
    process.env.PUBLIC_MONETIZATION_MODE === "adsense" ? "adsense" : "affiliate";
  return {
    mode,
    enabled: {
      // AdSense only renders when mode=adsense AND client ID is set.
      // Default "affiliate" means env leftovers can't accidentally surface ads.
      ads: mode === "adsense" && !!adsenseClient,
      amazon: !!amazonTag,
    },
  };
}

The default is "affiliate". If I forget to set the env var on a new deployment, AdSense doesn't accidentally appear and damage my publisher account reputation. To re-enable AdSense on ossfind when the quality work is done, it's one env var change in the Cloudflare Pages dashboard.

This is the same "safe default" principle I apply elsewhere in the stack — the post-deploy JSON-LD audit ensures broken structured data can't reach Google undetected; the monetization default ensures AdSense can't appear on a site that hasn't been approved.

I also added affiliate disclosure pages to both pivoted sites. The FTC requires disclosure when affiliate links appear; Amazon Associates adds its own ToS requirement. Each site now has /affiliate-disclosure with a footer link. The copy renders from the shared shared/legal package using a privacyPolicy(site, { ads }) function that switches between AdSense and affiliate text based on the monetization config. One source of truth for both modes.

What affiliate programs I'm actually using

For Top AI Tools:

Amazon Associates — primarily for AI-adjacent hardware (GPUs for local inference, books on practical ML) and tools that have physical product lines. Not every AI tool maps to an Amazon purchase, so this is supplementary coverage.
Direct SaaS programs — a handful of tools in the directory offer 20-30% recurring commission through their own partner programs. I'm applying to these individually. Slower to set up but higher per-conversion yield than Amazon.

For Find Games Like:

Humble Bundle Partner — covers Steam purchases through the Humble store. The commission on game sales is modest but consistent with audience behavior on a discovery site.
itch.io — no formal affiliate program. I link directly with no commission. Dropping itch games from the site to avoid the zero-commission awkwardness would be the wrong call; the indie-game audience expects to see itch alongside Steam.

I'm not using broad affiliate networks (CJ Affiliate, ShareASale) yet. At near-zero traffic, the compliance overhead isn't worth the incremental coverage. I'll add them when the sites hit meaningful monthly traffic volume — I'll know that threshold when I see it.

The falsifiable bet

By November 2026 — six months from launch — affiliate revenue on Top AI Tools and Find Games Like combined will exceed my estimate of what AdSense would have earned if approved on both sites.

My AdSense estimate is: display ad CPM on a new directory site (low traffic tier) × page views ≈ single-digit dollars per month per site in the early phase. Affiliate target: one to two conversions per month at modest commission values per conversion ≈ comparable range, with no approval delay.

The ranges overlap at low traffic. I'm not betting affiliate earns dramatically more. I'm betting it earns at least as much as AdSense would have, faster, without the approval lag and quality-work costs.

What would change my mind:

ossfind gets AdSense approved and earns substantially more per month than the other two sites combined via affiliate — that would signal the approval path has better unit economics than I modeled
A SaaS affiliate program rejects my application or adds compliance requirements that would distort editorial recommendations (I won't link to something I wouldn't recommend regardless of commission)
Traffic doesn't materialize on either site by month six — in which case the AI Overviews bet failed at a more fundamental level than the monetization question

The update I'll actually publish

I said in the initial architecture post that I'd publish real numbers at 30 and 60 days. That post is due in late May 2026 for the first set.

The metrics will include affiliate clicks and conversions broken down by site, AdSense quality-work progress on ossfind (measured by curated page count), and any Search Console signals worth sharing. I won't rationalize zero conversions as "still early" past month two. If the affiliate model isn't showing any signal by July, I'll say so and revisit.

The honest current state: affiliate earnings are $0 and AdSense is not running on any of the three sites. That's the baseline. Everything else is probability estimates based on the structural arguments above.

FAQ

Can affiliate programs earn anything at very low traffic?

Technically yes, but it's rounding error until you reach a few hundred monthly visitors from high-intent queries. At low single-digit monthly conversion rates, you need consistent traffic before the commission math produces anything worth reporting. This is why month-one data won't be meaningful — the same is true for AdSense. Both models need traffic.

Why not run both AdSense and affiliate on the same site?

AdSense policy allows affiliate links alongside display ads. But I'd rather keep ossfind as a clean AdSense application without the affiliate complexity for the reviewer to evaluate. Cleaner separation; easier to debug which factor drove any future rejection.

Can you switch back to AdSense on the pivoted sites later?

Yes. The implementation is one env var change. I specifically chose this pattern so no decision is permanent until the revenue data says it should be. If ossfind earns well under AdSense and the affiliate hypothesis turns out wrong, reversing either pivot is a ten-second config change.

Why keep ossfind on the AdSense track after two rejections?

The rejections were site-level, not account-level. The publisher account is in good standing. And the structural reason remains: the OSS-alternatives audience isn't buying — they're avoiding buying. Affiliate commission requires a purchase. Display ads monetize the visit. AdSense is the right model for that site if I can get the editorial quality to pass.

When do you expect to resubmit ossfind?

After I get the curated page count above 30. Currently at 18. Each nightly ETL run that generates real Claude Haiku content moves more entries across the threshold. I'm not setting a calendar date — I'll resubmit when the data supports it.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Three sleep intervals for three APIs: Steam 250ms, GitHub 100ms, HuggingFace none

MORINAGA — Wed, 10 Jun 2026 22:16:37 +0000

When I built the ETL pipelines for three programmatic directory sites in April — Top AI Tools (HuggingFace data), Find Games Like (Steam data), and Open Alternative To (GitHub data) — I had to figure out rate limits for three completely different APIs in the same week. The numbers, the failure modes, and the right way to handle errors are all different.

Here's what I actually shipped and the reasoning behind each number.

Steam: 250ms, deliberately aggressive

Steam's developer docs are sparse on hard rate-limit specifics. What I found from community discussion and trial: roughly 200 requests per 5 minutes per IP on the public Web API, which works out to one request per 1.5 seconds as a documented-safe interval. My code comments this openly:

await sleep(250); // Steam rate limit: ~200/5min, 1.5s is safe; 250ms is aggressive but usually fine

I chose 250ms anyway because the ETL runs as a nightly GitHub Actions job over ~60 game entries. At 250ms that's 15 seconds of sleep total. At 1.5 seconds it would be 90 seconds. The gap matters when the cron has three sites to process.

The acceptable risk: Steam doesn't hard-ban on the first rate-limit violation, it returns HTTP 429 and the job logs the error. The games ETL treats review-endpoint failures as non-fatal — the game row is still written; only the review stats are absent until the next run:

try {
  const r = await getAppReviewSummary(appid);
  // ... write to DB
} catch (err) {
  reviewsFailed++;
  console.error(`! Review fetch failed for appid ${appid}:`, err);
}

The reviewsFailed counter appears in the job log. If I see it climbing consistently, that's the signal to increase the sleep interval. So far I haven't needed to.

GitHub: 100ms, with authentication doing the real work

GitHub's REST API is explicit about limits: 60 requests per hour unauthenticated, 5,000 per hour with a personal access token. The GitHub docs on rate limiting cover both the primary limit and the secondary limits for specific endpoint categories. The OSS alternatives ETL makes one GET /repos/:owner/:repo call per alternative project — roughly 3–5 repos per SaaS tool in the seed data. Even a large seed run of 50 tools with 5 alternatives each is only 250 requests.

The sleep is there as a politeness interval, but authentication is doing the real rate-limit work:

function authHeaders(): Record<string, string> {
  const token = process.env.GITHUB_TOKEN;
  const base: Record<string, string> = {
    Accept: "application/vnd.github+json",
    "X-GitHub-Api-Version": "2022-11-28",
  };
  if (token) base.Authorization = `Bearer ${token}`;
  return base;
}

GITHUB_TOKEN is set in GitHub Actions from a repository secret. Without it, 60 requests per hour would exhaust in under a minute for a full seed run. With it, the 5,000/hour ceiling gives comfortable headroom.

One subtlety: there are two separate GitHub rate limits — the core REST API limit (5,000/hour authenticated) and the search API limit (30 requests per minute unauthenticated, 10 per second authenticated). The current ETL uses GET /repos/:owner/:repo directly, not search, so the looser core limit applies. If I ever switch to search-based discovery the math changes.

HuggingFace: no sleep, because none is needed

The model registry API — listing models, fetching model metadata — has no hard documented rate limit that I've hit in weeks of nightly runs. The ETL fetches up to 100 models in one GET /api/models?limit=100&sort=downloads call, then one detailed fetch per model. 100 rapid-fire requests, no sleep, no 429s.

Part of this is the HUGGINGFACE_TOKEN header in authenticated requests, which raises whatever ceiling exists. Part of it is that the registry API is explicitly designed for automated tooling at batch scale — it's the primary way model cards, metadata scrapers, and leaderboard tools consume the catalog.

function authHeaders(): Record<string, string> {
  const token = process.env.HUGGINGFACE_TOKEN;
  return token ? { Authorization: `Bearer ${token}` } : {};
}

If I scale to 1,000 models per nightly fetch I'd add a 50ms sleep as a precaution. For 100, the simplest thing that works is also the correct thing.

A comparison

API	Sleep	Auth impact	Failure mode	Fatal?
Steam appdetails	250ms	None (public)	429, occasional	Non-fatal
Steam reviews	250ms (shared)	None (public)	429, more frequent	Non-fatal
GitHub REST	100ms	60→5,000/hr	403, clear message	Non-fatal
HuggingFace registry	None	Raises ceiling	Rare 429	Non-fatal

All four code paths are non-fatal. A 429 or connection error anywhere in the batch writes a fallback-template row to Turso and increments a counter. The content upgrade loop picks up any gaps the next night.

The pattern that matters

The sleep interval is a guess. What actually protects the ETL from being useless after a rate-limit event is that failures are cheap. Every external API call in this stack is wrapped in a try/catch that writes degraded content rather than crashing the batch. The sleep interval controls how likely you are to hit a rate limit; the fallback chain controls what happens when you do.

For indie-scale ETL — tens to hundreds of entries per night — the combination of a conservative-ish sleep and a non-fatal error path is enough. If the site grows to thousands of entries per run, I'd rethink both: moving to a queue-bounded concurrent fetcher with exponential backoff, and separating the content generation from the data fetch into stages that can be retried independently.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

How I built a three-tier content quality ladder for programmatic directory ETL

MORINAGA — Tue, 09 Jun 2026 22:20:34 +0000

The three directory sites I launched in April — Top AI Tools, Find Games Like, and Open Alternative To — all generate editorial content the same way: fetch metadata from an external API, send it through Claude Haiku 4.5, write the result to Turso. But that description skips the part that actually matters for a programmatic site at scale: what happens when Claude can't run.

The answer is a content quality ladder with three tiers, tracked by a single model_used column.

The three tiers

Every content table across all three sites has a model_used column. It takes one of three values:

Value	Origin	Quality
`seeded-from-json`	Loaded from a curated JSON file at bootstrap	Minimal — structured but thin
`fallback-template`	Claude unavailable or API key absent	Acceptable — technically correct, not editorial
`claude-haiku-4-5`	Generated by Claude Haiku 4.5	Target — editorial summaries, named examples, nuanced caveats

Seeded content exists because each site ships with a JSON file of curated entries. Those entries have names, descriptions, and metadata from their upstream source (HuggingFace, Steam, GitHub), but no editorial layer yet. The page renders — but it reads like a database dump, not a directory.

Fallback-template content is what you get when the API key isn't present or when a Claude call fails. For the AI tools site, the fallback for a model named qwen2-7b in the text-generation pipeline looks like this:

qwen2-7b is an open-source text-generation model available on HuggingFace.
Details are sourced from the public model registry.

That's not wrong. It just doesn't help anyone decide whether to use the model.

Claude Haiku content is the target state. A good generation for the same model says something like: "Qwen2-7B is a 7-billion parameter instruction-tuned model from Alibaba Cloud optimized for multilingual generation, showing strong performance on Chinese and English benchmarks while fitting in 16GB of VRAM." The difference is editorial voice and specificity — neither of which template-filling can produce.

The upgrade query

The ETL generation step doesn't blindly regenerate everything on each run. It targets only entries that need work:

SELECT m.id, m.name, m.pipeline_tag, m.tags
FROM models m
LEFT JOIN model_content c ON c.model_id = m.id
WHERE c.model_id IS NULL
   OR c.model_used IN ('fallback-template', 'seeded-from-json')
ORDER BY m.downloads DESC
LIMIT ?

Three things happen simultaneously here:

LEFT JOIN ... WHERE c.model_id IS NULL catches brand-new entries added by the nightly fetch that have no content row yet.
OR c.model_used IN ('fallback-template', 'seeded-from-json') catches existing rows that were written with lower-quality content.
ORDER BY m.downloads DESC means when the LIMIT is hit, the most-downloaded (most-visited) entries are upgraded first.

This identical query pattern appears in all three sites with different table names: models/model_content for AI tools, games/game_content for indie games, saas/saas_content for OSS alternatives. The abstraction was a late realization — I wrote it three times before noticing it was the same thing. A shared buildUpgradeQuery(tableName, pkField, contentTable) helper would have been the right call from the start.

The fallback chain

Inside the generation loop, every entry goes through the same decision tree:

const hasApiKey = !!process.env.ANTHROPIC_API_KEY;

if (hasApiKey) {
  try {
    const result = await generate({
      systemPrompt: SYSTEM_PROMPT,
      userPrompt,
      cacheSystem: true,
      maxTokens: 1024,
    });
    content = parseOrFallback(result.text, fb);
    modelUsed = "claude-haiku-4-5";
    generated++;
  } catch (err) {
    console.error(`! Claude error for ${id}:`, err instanceof Error ? err.message : err);
    content = fb;
    fallback++;
  }
} else {
  content = fb;
  fallback++;
}

The cacheSystem: true flag marks the system prompt block with cache_control: { type: "ephemeral" }. All three sites have fixed system prompts — the same AI tools instruction across every model generation, the same game critic instruction across every game — so the first call in a batch primes the cache and the remaining ~99 calls read it at the reduced input rate. I covered the mechanics in the article on the shared Haiku client. With a ~900-token system prompt and 100 entries per run, the cache saves roughly 90,000 input tokens per nightly run. Anthropic's prompt caching documentation has the exact pricing for cache creation vs cache read tokens.

The error path is deliberately non-throwing. Any Claude failure — rate limit, network timeout, malformed response — drops through to content = fb and increments fallback. The run continues. If 10 of 100 Claude calls fail due to transient rate limits, 90 get written with claude-haiku-4-5 and the 10 failures get fallback-template. Those 10 rows surface in the next night's upgrade query automatically.

The upsert write

Every content row is written with INSERT ... ON CONFLICT ... DO UPDATE SET:

INSERT INTO game_content
  (appid, summary, similar_games, good_for, avoid_if, generated_at, model_used)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(appid) DO UPDATE SET
  summary = excluded.summary,
  similar_games = excluded.similar_games,
  good_for = excluded.good_for,
  avoid_if = excluded.avoid_if,
  generated_at = excluded.generated_at,
  model_used = excluded.model_used

The upsert makes the ETL fully idempotent: running it twice produces the same state as running it once. More importantly, it means the model_used column gets overwritten when an upgrade succeeds. A row that was fallback-template becomes claude-haiku-4-5 in-place, without any explicit "mark upgraded" step. The column just reflects what actually produced the current content.

The compare-page ETL uses a different pattern: check-before-insert with an explicit SELECT 1 to skip already-generated pairs. Both patterns are valid. Check-before-insert is better when reprocessing is expensive (large Claude calls, multi-step generation). Upsert-overwrite is better when you always want the latest generation to win regardless of what was there before.

The noindex safety valve

One consequence of shipping a three-tier system is that some pages launch with genuinely thin content. For the indie games site, the threshold is explicit in the game page component:

const noindex =
  game.good_for.length === 0 &&
  game.avoid_if.length === 0 &&
  game.similar_games.length === 0;

If a game entry has no good_for audience signals, no avoid_if caveats, and no similar game suggestions — which happens when the content row is missing entirely, not just fallback-template — the page gets noindex in its robots meta. The page renders fine for direct visitors; it just isn't submitted to Search Console until content exists.

In practice, the fallback templates do populate good_for and avoid_if with generic strings like "Indie game enthusiasts" and "You prefer AAA production values," so most fallback-template entries still pass the noindex check. The valve fires mainly on completely-missing rows, which are brief windows between when the fetch ETL adds a new game and when the generation ETL runs next.

The export step

After generation, a separate export.ts script dumps the content tables to static JSON files that Astro reads at build time. This is the architectural detail that makes the quality ladder safe to run asynchronously.

If the Anthropic API is down for an entire nightly run, the export runs with whatever's in the DB, the Astro build succeeds with existing content, and the deployed site doesn't have zero-content pages. The upgrade queue just has a larger backlog the following night.

The static SSG approach I'm running across all three sites is partly justified by this property. Dynamic rendering from a live DB would mean a Claude outage or Turso blip directly impacts page load time for real users. The ETL → export → build pipeline adds ~24 hours of content staleness in exchange for availability that doesn't depend on the API being up at request time. For a directory site where model descriptions change rarely, that tradeoff is easy to accept.

What I'd do differently

The generation loop is strictly sequential. One call, await, write to DB, next entry. For 100 entries at roughly 1–1.5 seconds per call that's about 2 minutes per run — fine for the current scale.

At 1,000 entries it would be 20+ minutes, which starts blocking the rest of the GitHub Actions job. The fix is a semaphore-bounded batch:

import PQueue from "p-queue";
const queue = new PQueue({ concurrency: 5 });

const tasks = pending.rows.map((row) =>
  queue.add(() => generateAndWrite(row))
);
await Promise.all(tasks);

Five concurrent workers would bring a 1,000-entry run down to under 5 minutes without risking the Anthropic rate limit. I've kept the sequential version because it's simpler to debug and the current batch sizes don't need it, but I'll add the queue before growing any site past ~300 entries.

I also wish I'd started with better fallback copy. The initial seed templates are technically correct but thin, and some of that thin content shipped live to indexable pages before the ETL had a chance to upgrade it. A cleaner v1 strategy: run the full ETL before the first Astro build so every page that ships has at least a real Claude generation. The seeded-from-json tier exists because I moved too fast at launch; it's not architecturally necessary.

FAQ

Can I run the ETL without an API key during local development?

Yes. The hasApiKey check means every generation falls through to fallback-template. All DB writes still happen, the export still runs, and the Astro build succeeds. Once you add a real key, the next ETL run upgrades all fallback-template rows automatically without any manual intervention.

How do I check the current upgrade ratio?

SELECT model_used, COUNT(*) as cnt
FROM game_content
GROUP BY model_used;

A healthy site a week after launch should have mostly claude-haiku-4-5 rows with fallback-template count trending toward zero. The generated_at timestamp on each row also lets you see how recently content was last upgraded.

What happens when Claude returns malformed JSON?

Each site's parseOrFallback() function extracts the outermost {...} block with a regex before parsing — this handles the common case where Haiku prepends an explanation like "Here is the entry:" before the actual JSON. All field accesses after the parse are null-safe and fall back to the fallback struct individually if a field is wrong type or missing. The row still gets written; model_used records whichever tier actually filled the content.

Does the cache persist between separate nightly runs?

No. Anthropic's ephemeral cache TTL is 5 minutes. Within a single run of 100 entries, the 99 calls after the first hit the cache. Across runs scheduled hours apart, the cache has expired and the first call re-primes it. The savings are per-batch, not cross-run — still meaningful for batches of 100, but not a persistent cost reduction over time.

Why Turso for this instead of Postgres?

I covered the comparison in detail in the Turso vs Cloudflare D1 article. The short version for this use case: @libsql/client works identically in Node.js ETL scripts and at Astro serverless/edge, with no separate driver or connection-pooling setup for each environment. For a project where the same getClient() call needs to work in GitHub Actions jobs and Vercel edge functions, that's the practical reason to use it.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Static site search for Astro in 2026: why I picked Pagefind over Algolia and Lunr

MORINAGA — Tue, 09 Jun 2026 22:19:50 +0000

I added search to all three of my AI-curated directory sites last month. The choice wasn't obvious — there are at least four options with real adoption — so here's the breakdown I actually ran through before landing on Pagefind.

The four options I considered

Pagefind is a Rust-based static search library. It runs at build time, generates an index in /_pagefind/, and serves everything as static files. No backend, no API key, no per-query billing. It ships a prebuilt UI (PagefindUI) that you can mount on any element, and it supports WebAssembly for in-browser querying.

Algolia DocSearch is free for open-source documentation sites, $49/month for commercial sites below a certain crawl limit. It indexes your content via their crawler (or an API push), stores it on Algolia's infrastructure, and gives you a hosted search widget. Fast, polished, and battle-tested — it's what most major docs sites use.

Lunr.js is a client-side search library. You build the index at build time, serialize it to JSON, and ship it with the page. The browser loads the entire index on first search. Works offline, no external dependency, but the index size grows linearly with content, and there's no incremental loading.

FlexSearch is a newer alternative to Lunr with better performance characteristics and smaller bundle size, but the same core trade-off: you ship the whole index to the browser upfront.

Why Pagefind won

The decisive factor was index size management. My directories have 500-1,000 entries per site, each with a multi-paragraph generated description. A Lunr index for 1,000 entries would be 2-4MB shipped with every page load. Pagefind shards its index and loads chunks lazily as the user types — so the initial load is under 30KB (the WASM binary + a small manifest), and individual chunk fetches happen on demand.

The second factor was cost. Algolia DocSearch's commercial tier runs $49/month per site. I'm running three sites on a total infrastructure budget of roughly $25/month. Pagefind is free.

The third factor was the deploy model. Because everything in /_pagefind/ is a static file, Cloudflare Pages caches it at the edge with no configuration. There's no API to rate-limit, no service availability to depend on, no API key to rotate.

The SearchDialog implementation

The search component is a <dialog> element with a Pagefind UI mounted inside it. I load the pagefind-ui.js script lazily — only when the dialog is first opened — to keep it off the critical path:

function loadPagefind() {
  if (loaded || !root) return;
  loaded = true;
  var s = document.createElement("script");
  s.src = "/_pagefind/pagefind-ui.js";
  s.onload = function () {
    if (window.PagefindUI) {
      new window.PagefindUI({ element: root, showSubResults: true, resetStyles: false });
    }
  };
  s.onerror = function () {
    root.innerHTML = '<p>Search index not available yet (first build). Try again after next deploy.</p>';
  };
  document.head.appendChild(s);
}

The s.onerror handler is the part most tutorials skip. On the first deploy of a new Cloudflare Pages site, the /_pagefind/ directory doesn't exist yet — Pagefind only runs during the build. If a user opens search before the first full build completes, pagefind-ui.js 404s. Without the error handler, you get a silent failure. With it, you get a legible message.

The <dialog> element is the right primitive here: it handles focus trapping automatically, Escape closes it natively, and backdrop: CSS pseudo-element gives you the dimmed overlay without JavaScript. The Cmd+K keyboard shortcut is wired with document.addEventListener("keydown", ...) — no library needed.

What Pagefind doesn't do

Two gaps I've hit:

No query logging. Pagefind runs entirely in the browser and doesn't send queries anywhere. For a commercial directory, knowing what users search for is valuable — it tells you which models or games to add, and which compare pages to prioritize. With Algolia you get this for free. With Pagefind you'd need to add a thin logging layer (a fetch POST to an analytics endpoint on each query event). I haven't built this yet.

No fuzzy matching out of the box. Pagefind does stemming and basic substring matching, but "stabilty diffusion" (typo) won't match "stable diffusion". Algolia's typo-tolerance is significantly better. For an AI tools directory where model names are long and often misremembered, this matters. I'll probably add a query-suggestion layer that does fuzzy pre-matching before handing off to Pagefind.

Quick comparison table

	Pagefind	Algolia DocSearch	Lunr.js
Cost	Free	$49/mo (commercial)	Free
Index location	Static files	Algolia cloud	Shipped with page
Initial JS load	~30KB	~80KB	~10KB + index
Index size scalability	Chunked, lazy	Server-side	Linear, upfront
Typo tolerance	Basic stemming	Strong	Weak
Query logging	No	Yes	No
Build-time integration	Yes	Crawler / push API	Yes

For a static site on a tight infrastructure budget with 500-1,000 entries, Pagefind is the right default. If the site were larger or if I needed typo tolerance and query analytics without building them myself, Algolia would be worth the cost.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

How I built pairwise AI model compare pages with Claude Haiku and a budget cap

MORINAGA — Tue, 09 Jun 2026 22:19:37 +0000

When I added compare pages to the Top AI Tools directory, the first question I had to answer was: how many pairs am I actually looking at? With roughly 200 models across 8 pipeline tags, the naive upper bound is 200 × 199 / 2 ≈ 19,900 pairs. Generating content for each one with Claude Haiku would cost somewhere around $20 per run — not ruinous, but not something I wanted to run daily without thinking carefully.

Here's what I actually built, where it falls short, and what I'd do differently if starting over.

The combinatorics problem

Model compare pages exist for a specific type of query: "llama 3 vs mistral 7b", "stable diffusion vs sdxl", "whisper vs wav2vec2". These are high-intent queries — the user has already narrowed down to a shortlist and wants a concrete decision nudge. The static SSG approach I'm running means I need to precompute each compare page at build time, which puts pressure on how many pages I can afford to generate.

The solution I landed on: group by pipeline_tag, pair the top-4 models by download count within each group, then cap total pairs with a COMPARE_LIMIT env var. Within a single pipeline like text-generation, the top 4 models give 6 pairs (4 choose 2). Across 8 active pipelines that's roughly 48 pairs. The env cap of 50 means I stay within that budget while having room to grow.

const byPipe = new Map<string, typeof models>();
for (const m of models) {
  if (!m.pipeline_tag) continue;
  const arr = byPipe.get(m.pipeline_tag) ?? [];
  arr.push(m);
  byPipe.set(m.pipeline_tag, arr);
}

const pairs: Array<[Model, Model]> = [];
for (const [, list] of byPipe) {
  const sorted = [...list].sort((a, b) => b.downloads - a.downloads);
  const take = sorted.slice(0, Math.min(4, sorted.length));
  for (let i = 0; i < take.length; i++) {
    for (let j = i + 1; j < take.length; j++) {
      pairs.push([take[i]!, take[j]!]);
    }
  }
}
const chosen = pairs.slice(0, MAX);

The pairing happens entirely within pipelines right now, which means I'm covering "llama vs mistral" (both text-generation) but not "whisper vs gemini-vision" (cross-pipeline). Cross-pipeline comparisons are actually more valuable for users who don't know the landscape yet — that's the next iteration.

The pair_slug and idempotent inserts

The slug for each compare pair is constructed deterministically: sort the two model slugs alphabetically, join with --vs--. So whether the ETL processes (llama-3, mistral-7b) or (mistral-7b, llama-3), the slug is always llama-3--vs--mistral-7b.

const pairSlug = [a.slug, b.slug].sort().join("--vs--");

This makes the entire ETL idempotent. The script runs every night. If all pairs already exist in the DB, it exits in a couple of seconds without a single Claude call. I check before inserting rather than using INSERT OR IGNORE at the SQL level — the explicit check lets me count skipped vs generated in the same run, which I log:

[compare] done — generated: 3, skipped: 47

This matters for monitoring. A run that generates 0 and skips 50 is healthy. A run that generates 0 and skips 0 (nothing in DB, nothing processed) would indicate a bug.

Claude Haiku with system-prompt caching

I reuse the shared Haiku client I built in week one, which handles cacheSystem: true on the system prompt. Since the system prompt — the JSON schema instruction — is identical across all compare calls, the first call primes the cache and subsequent calls see near-zero token cost on that prefix.

The user prompt includes both model names, their authors, pipeline tags, and up to 400 characters of their existing summaries (which come from the earlier content generation step):

const userPrompt = `Compare these two AI models:
A: ${a.name} (author: ${a.author ?? "unknown"}, pipeline: ${a.pipeline_tag ?? "unknown"})
   Summary: ${a.summary?.slice(0, 400) ?? "(none)"}
B: ${b.name} (author: ${b.author ?? "unknown"}, pipeline: ${b.pipeline_tag ?? "unknown"})
   Summary: ${b.summary?.slice(0, 400) ?? "(none)"}

Produce the JSON comparison.`;

Truncating summaries at 400 characters keeps the user prompt lean. Compare pages are about the delta between two models, not a rehash of each model individually. I already have dedicated model pages for depth; the compare page needs to answer "which one, for what" — that takes maybe 6 sentences total.

The system prompt requests a JSON object with summary, differences (array), similarities (array), and recommendation. Keeping the output shape narrow means Haiku rarely wanders off-schema.

JSON parsing with a regex fence

Even with tight prompting, Haiku occasionally produces JSON with an explanation preamble: "Here is the comparison:" followed by the actual object. Strict JSON.parse on the raw output would throw. I extract the outermost {...} block with a regex before parsing:

function parseCompare(text: string, fb: CompareData): CompareData {
  try {
    const m = text.match(/\{[\s\S]*\}/);
    if (!m) return fb;
    const p = JSON.parse(m[0]);
    return {
      summary: typeof p.summary === "string" ? p.summary : fb.summary,
      differences: Array.isArray(p.differences)
        ? p.differences.map(String)
        : fb.differences,
      similarities: Array.isArray(p.similarities)
        ? p.similarities.map(String)
        : fb.similarities,
      recommendation:
        typeof p.recommendation === "string"
          ? p.recommendation
          : fb.recommendation,
    };
  } catch {
    return fb;
  }
}

Each field is validated individually before being accepted. If differences comes back as a string (occasional Haiku behavior when it conflates the array with a comma-separated list), the page falls back to the template for that field rather than crashing.

The fallback struct is worth writing carefully. I spent five minutes on mine and it shows:

const fb: CompareData = {
  summary: `${a.name} and ${b.name} are both ${a.pipeline_tag} models. See each entry for specifics.`,
  differences: ["See individual model pages for architecture and use cases."],
  similarities: ["Both are open-source models on HuggingFace."],
  recommendation: "Pick based on your compute budget and specific task requirements.",
};

A user landing on a fallback-generated compare page gets a technically-true page that directs them to the model pages rather than a blank or error state. The model_used column in the DB records "fallback-template" for these rows, which I use to identify candidates for regeneration.

Storage in libSQL and the static JSON dump

Compare data lives in a model_compare table in Turso libSQL, with a unique constraint on pair_slug. After the ETL loop, everything gets dumped to compare.json for the static build:

const all = await db.execute(
  `SELECT * FROM model_compare ORDER BY slug_a, slug_b`
);
const entries = all.rows.map((r) => ({
  slug_a: String(r.slug_a),
  slug_b: String(r.slug_b),
  pair_slug: String(r.pair_slug),
  summary: r.summary ? String(r.summary) : "",
  differences: r.differences ? JSON.parse(String(r.differences)) as string[] : [],
  similarities: r.similarities ? JSON.parse(String(r.similarities)) as string[] : [],
  recommendation: r.recommendation ? String(r.recommendation) : "",
}));
await writeFile("./src/data/compare.json", JSON.stringify(entries, null, 2));

The Astro build reads this JSON at build time, generating one static page per pair. No runtime DB calls, no cold starts. The tradeoff is freshness: compare content is up to 24 hours stale. For "llama 3.1 vs llama 3.2", that's fine — the models don't change daily.

I validate the JSON-LD on compare pages through the post-deploy audit CI step the same way I do for individual model pages. Structured data matters more on comparison queries because those are the exact queries that AI Overviews tend to surface, so getting the schema right is worth the CI overhead.

The Astro slug generation for compare pages uses the pair_slug directly. The URL pattern is /compare/llama-3--vs--mistral-7b/, which is ugly but unambiguous — the double-dash separator makes it clear this is a two-part slug rather than a hyphen in a model name.

What I'd change starting over

Generate cross-pipeline pairs from day one. The most useful compare queries aren't "llama 3.1 vs llama 3.2" — users who care about that distinction already know. The interesting queries are cross-category: "should I run inference on a text-generation model or use a RAG pipeline?" I skipped this to stay within the budget cap, but it means I'm missing the long-tail traffic that would actually be differentiated from generic model pages.

Drive pair selection from search query logs. Right now I pick pairs by download rank. A better signal would be which pairs users actually search for. Pagefind runs client-side and doesn't log queries to any server, so I'd need a thin logging endpoint — something like a POST to a GitHub Actions-triggered function that appends to a JSONL file. Then the ETL reads the top-N ungenerated pairs from the log. This is a small amount of infrastructure but it would make the pair selection much more demand-driven.

Raise the budget cap. MAX=50 is conservative. At current Haiku pricing with prompt caching, 500 pairs would cost roughly $0.10 per nightly run. I was cautious when I set the default, but I've watched the billing closely and the actual spend is a fraction of what I modeled. I'll bump this to 200 in the next ETL config update.

The itch.io entries pattern I added to the indie-games directory taught me to plan for the second data source earlier. Compare pages have the same shape: a join between two rows. Getting the abstraction right before you have 500+ rows in the DB is much easier than retrofitting it.

FAQ

Does the ETL run every night even when no new models are added?

Yes, but it's nearly free when nothing is new. The check-before-insert means most nights it does 50 DB reads and exits in under 3 seconds without touching the Claude API. The console output shows generated: 0, skipped: 47 which is the signal that everything is up to date.

What happens when Claude returns malformed JSON?

parseCompare catches the error and returns the fallback struct. The row is still written to the DB with model_used = "fallback-template", which I can query to find rows worth retrying. In practice, this happens on maybe 2-3% of generations — usually when the two models have very sparse metadata and Haiku doesn't have enough context to produce structured output.

Does the compare.json file get unwieldy as pairs accumulate?

At 50 pairs it's roughly 25KB. At 500 pairs it'll be around 250KB — still fine for build-time loading in Astro. If I ever hit 5,000 pairs I'd split the file by pipeline_tag and lazy-import only the relevant subset for each page. For now, one flat JSON file is simpler and fast enough.

Why not compute compare content at request time with an edge function?

Cold starts and cost. An edge function hit for each compare page view would add 200-500ms of latency (Haiku inference + DB round trip) and would cost much more per-pageview than the nightly batch approach. The content also doesn't need to be fresher than daily — model capabilities don't shift on an hourly basis. Static precomputation is the right tradeoff here, consistent with the broader bet on static SSG I'm running on all three sites.

How do you handle the case where a model is removed from HuggingFace?

Right now, I don't. If model foo is deleted from HuggingFace but its compare rows are still in the DB, those compare pages will still be served at build time. They'll have the old data until the model's row in models.json is removed — which only happens if the model falls out of the top-500 in the nightly fetch. It's a known gap. For now, the risk is low; popular models don't disappear. A more robust system would cross-reference the compare table against the model table and tombstone orphaned pairs.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Five overlooked packages running my AI directory stack

MORINAGA — Tue, 09 Jun 2026 03:54:02 +0000

The interesting parts of a project are not always the AI model or the hosting platform. This week I spent time reading source code for five dependencies that sit quietly in my package.json files. None of them are trending. All of them are load-bearing.

My stack is Astro 5 SSG + Turso libSQL + GitHub Actions cron + Claude Haiku 4.5. Three sites: Top AI Tools, Find Games Like, Open Alternative To. Seven weeks in, still under 400 total pageviews, but the infrastructure is solid enough that I can focus on content rather than firefighting.

tsx — TypeScript without the build ceremony

tsx by Hiroki Osame is how I run every ETL script in the monorepo. The command tsx src/etl/run.ts just works — no tsconfig fiddling, no ts-node --esm flags, no separate compile step. Under the hood it uses esbuild, which means startup is fast enough that a five-second cron warm-up doesn't matter.

What surprised me when I read the repo: tsx strips types with esbuild rather than the TypeScript compiler, so it doesn't type-check. That's intentional. For ETL scripts where I want pnpm typecheck to catch structural errors at CI time but not slow down the hot path, this is exactly the right tradeoff. The README calls this out clearly. I wish I'd read it three weeks ago instead of assuming tsx did full type checking.

Pagefind — static full-text search with no server

Pagefind runs as my postbuild step: pagefind --site dist --output-subdir _pagefind. It crawls the built HTML, creates a compressed WASM index, and the client-side JS loads only the chunk it needs per query. The result is search that works on a static Vercel or Cloudflare Pages deploy with zero additional infrastructure.

I read through the index format docs this week. The segment files are stored as zstd-compressed binary blobs, and the JS client fetches them lazily based on the query prefix. For three sites each under 2,000 pages, the index stays under 500 KB total. The PageFind UI component is optional — I replaced it with a plain <input> that calls the JS API directly so I could control the result rendering in Astro components.

Crawlee — TypeScript scraping with built-in queue management

I haven't shipped Crawlee yet, but it's been on my bookmarks list since I started building the itch.io ETL. My current approach is fetch + manual parsing, which works for known endpoints. Crawlee adds request queue persistence, rate limiting, and a cheerio integration for HTML extraction, all in TypeScript with native ESM support.

The reason I haven't switched: my ETL runs inside GitHub Actions where I want simple, auditable scripts over a full crawl framework. But if I start scraping product pages from sites that don't have APIs — which is the next natural expansion for the OSS alternatives directory — Crawlee is the tool I'd reach for. The Apify team maintains it actively and the TypeScript types are genuinely good.

eemeli/yaml — small footprint, strict spec compliance

The yaml package by Eemeli Aro parses the frontmatter in my article files before cross-posting to Dev.to and Hashnode. It's 35 KB minified, has zero dependencies, and handles multi-line strings and nested objects without surprises. I switched from js-yaml six weeks ago because eemeli/yaml has better ESM exports and the parse errors are more actionable when frontmatter has a typo.

One thing I didn't know until this week: the yaml package can also stringify back to YAML, preserving comments. I don't use that feature yet, but it matters for a workflow where I want to programmatically update article frontmatter without clobbering the human-readable structure. That's on the roadmap for automating canonical_url injection after Dev.to publish.

@libsql/client — batched writes are the underrated feature

The @libsql/client TypeScript client is what connects my ETL scripts to Turso. I wrote about Turso vs Cloudflare D1 earlier this week, but I didn't cover the batch API, which is the feature I actually rely on most. A single db.batch([...]) call wraps multiple INSERT OR REPLACE statements in one network round trip, which matters when seeding a 500-row table from a GitHub Actions runner.

The client supports both remote Turso connections and an embedded file: mode that runs libSQL in-process with no network. I use the in-process mode for local ETL development so I don't burn Turso API quota while iterating on the seed logic. Switching between modes is one environment variable. That's the kind of DX detail that makes a dependency feel considered rather than assembled.

None of these packages announced anything dramatic this week. They're just the boring infrastructure that lets the AI parts of the stack do their job. I'll write up actual traffic and content metrics in 30 days when I have a month of data worth publishing.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Five things that caught my attention this week in AI tools and open-source models

MORINAGA — Tue, 09 Jun 2026 03:53:19 +0000

A lighter week for me operationally — content refreshes, a YouTube analytics update, some Bluesky queue maintenance. Which meant more time to actually read things. Here are five items that stuck.

1. Claude Code Agent View changes the mental model

Anthropic shipped Agent View inside Claude Code on May 11. It's a unified dashboard for managing multiple parallel Claude Code sessions: start a session, send it to the background, check results when you want to. The interface treats individual sessions the way a CI dashboard treats builds.

I've been running Claude Code by opening multiple terminals with different working directories. It works, but the overhead of context-switching between tabs adds up fast. A UI that surfaces what each agent is doing without requiring a terminal switch is more than quality-of-life — it shifts Claude Code from "smart terminal" to "orchestration layer."

That's the direction I think AI coding tools are heading. The question isn't whether you can have a useful conversation with an AI about code. It's whether you can queue up a batch of distinct tasks, step away, and come back to something actionable. Agent View is an early answer to that question.

2. ZAYA1-8B trained on AMD hardware is a supply chain signal

Zyphra released ZAYA1-8B under Apache 2.0 around May 6-7. It's a mixture-of-experts architecture: ~8B total parameters, ~760M active per token. Standard MoE efficiency math. What's not standard: the entire training run used AMD Instinct hardware.

The serious open-weights training runs are almost universally done on NVIDIA H100s or A100s. Zyphra shipping a competitive reasoning model that's clean Apache license and trained end-to-end on AMD is a concrete counter-example to "you need NVIDIA to train anything worth using."

That doesn't mean AMD is catching up fast enough to matter at scale yet, or that my next fine-tune would go faster on Instinct hardware. It means the GPU monoculture in open-source training has a verifiable crack in it. I'm watching whether other small labs follow.

3. The Harness productivity report has a buried lede

Harness released The State of Engineering Excellence 2026 on May 13. The headline: 89% of engineering leaders report improved developer productivity; 88% report improved satisfaction since adopting AI coding tools.

The headline is predictable. Every vendor survey about AI tools says the same thing. The part worth reading is the buried finding: AI has outpaced the measurement frameworks organizations use to track productivity. Existing DORA metrics — deployment frequency, change failure rate, MTTR, lead time — weren't designed for workflows where a human is reviewing and steering AI-generated output rather than writing from scratch.

If you're building dev tooling and trying to sell to engineering leaders right now, "AI made us faster" is table stakes. "Here's what to measure instead, and here's how we surface it for your team" is the actual product bet worth making.

4. ServiceNow Build Agent went GA inside Claude Code and Cursor

ServiceNow announced on May 13 that Build Agent is generally available in ServiceNow Studio and extended its core skills into Claude Code, Cursor, Windsurf, and GitHub Copilot — with governance defaults on. Developers can build with ServiceNow APIs from their own editors without leaving their environment.

The governance-by-default choice is the interesting design decision here. Most IDE integrations hand full control to the developer and assume IT will configure guardrails separately. ServiceNow's bet is that enterprise buyers want the platform's access controls and audit trails to travel with the tool automatically. Harder to sell on a feature list; better moat if the bet holds.

5. I removed MCP servers from my pipeline and reliability went up

This one is personal. I dropped several MCP server connections from my content pipeline this week (the commit message is "i-removed-mcp-servers-and-my-pipeline-got-more-reliable," which about covers it).

MCP servers add real capabilities. They also add failure surfaces: network timeouts, schema drift when a remote API changes without warning, authentication tokens that expire silently at 3 AM. My ETL runs unattended on a cron schedule. When a remote MCP call hangs, the whole job hangs. I didn't always know until I checked results the next morning.

The lesson I'm taking: MCP integrations are excellent for interactive sessions where a human is watching and can handle a failure gracefully. For scheduled, unattended workflows, each external dependency is a reliability tax you pay whether or not you're awake to collect it. I'm keeping MCP for interactive use and building local fallback paths for anything production-critical.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Turso libSQL vs Cloudflare D1 for an Astro monorepo: the practical difference

MORINAGA — Tue, 09 Jun 2026 03:53:06 +0000

When I set up the shared ETL database for three Astro SSG directory sites, I had two obvious SQLite-at-the-edge options: Turso (libSQL, runs anywhere) and Cloudflare D1 (SQLite inside Workers). I went with Turso. Here's the practical difference that drove the decision.

The local dev problem D1 doesn't solve cleanly

Cloudflare D1 is native to the Workers runtime. If you're using Cloudflare Workers for server-side rendering, D1 is the obvious choice — it's edge-collocated with zero config and the env.DB binding is automatic.

My setup is different. The sites are static Astro 5 SSG on Cloudflare Pages — no Workers, no server runtime. The ETL pipeline that populates the database runs in GitHub Actions. Nothing in my stack executes inside the Workers environment.

To use D1 from GitHub Actions you either use the Cloudflare REST API or the wrangler CLI. Both work, but neither gives you a local SQLite file you can query directly during development. You'd be hitting a remote database for every SELECT during local ETL runs or testing schema changes. Wrangler does have a --local flag that writes to a local D1 file, but the path and format differ from the production D1 setup, so you're managing two different code paths.

Why Turso's local fallback changes the calculus

The @libsql/client package accepts a url that can be either a libsql:// remote URL or a file:// path:

export function getClient(): Client {
  if (cached) return cached;
  const url = process.env.TURSO_DATABASE_URL ?? "file:./data/local.db";
  const authToken = process.env.TURSO_AUTH_TOKEN;
  cached = createClient({ url, authToken });
  return cached;
}

In CI, TURSO_DATABASE_URL is set to the Turso remote URL. On my laptop, the variable isn't set, so the client opens file:./data/local.db — a plain SQLite file on disk. Same @libsql/client package, same query API, same schema. The code path is identical.

This means I can run ETL scripts locally and inspect the database with any SQLite viewer. Schema migrations apply with the same applyMigrations() call used in production:

export async function applyMigrations(
  migrations: readonly string[]
): Promise<void> {
  const client = getClient();
  for (const sql of migrations) {
    await client.execute(sql);
  }
}

No Docker containers. No Wrangler flags. No separate local-vs-remote code path. The same SQL that creates the models table locally creates it in Turso.

What the migration pattern actually looks like

Each app defines its own migration array. The run.ts entrypoint for the AI tools ETL calls applyMigrations([CREATE_MODELS_TABLE, CREATE_REVIEWS_TABLE, ...]) at startup. If the table already exists, CREATE TABLE IF NOT EXISTS is a no-op. Idempotent, no migration runner needed.

This is the same philosophy as the ETL publish step — the article publish pipeline checks published_urls in the frontmatter before posting, so re-running never double-posts. The database migration check follows the same pattern: check-then-act, idempotent by design.

Where D1 would actually win

If any part of the stack were running inside Cloudflare Workers — a search endpoint, an API route, a middleware layer — D1 would be the stronger choice. The env.DB binding is faster than a network call to Turso's edge, and you don't need to manage auth tokens for a same-datacenter query.

My architecture is fully static because the freshness vs. speed trade-off works in SSG's favor for directory content. No Workers means D1's core advantage doesn't apply.

If I add a Cloudflare Worker for site search or a revalidation webhook, I'd reconsider. A hybrid Turso (ETL reads/writes in GitHub Actions) + D1 (Workers queries) setup would be more complex than I want right now.

Three things I don't know yet

Concurrent write performance. The ETL pipeline has max-parallel: 1 set explicitly in the workflow. Writes are serial and controlled. I haven't tested what happens with concurrent writes, and Turso's concurrent write behavior on the free tier is not something I've stress-tested. I'll know more in 30 days.

Migration safety at schema evolution. Adding a nullable column to an existing table is straightforward. Renaming a column or changing a type requires a table rebuild. I haven't had to do either yet. When I do, the applyMigrations() approach will require careful ordering.

D1 pricing at scale. Turso's free tier covers 500 databases and 1 billion row reads per month. Cloudflare D1's free tier is similar. At the current scale of three sites with daily ETL runs, neither would cost anything. If traffic grows enough to matter, I'll publish actual numbers — not estimates.

The database choice was not the interesting part of this project. The local file fallback is the entire reason I made it; everything else is roughly equivalent between the two for my use case.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

What I learned generating OG images for articles with Playwright and zero API cost

MORINAGA — Tue, 09 Jun 2026 03:52:22 +0000

The conclusion first: for a batch of under a few hundred static articles, generating OG images by screenshotting HTML templates with Playwright costs nothing, gives you full CSS control, and requires zero external API keys. The trade-offs are real — it's slow per image, it's not suitable for on-demand generation, and it has a hidden dependency on network availability during the build step. But for my use case, those trade-offs don't hurt.

Here's how the script works, what broke, and what I'd do differently.

Why I avoided image generation APIs

My three directory sites — aiappdex.com, findindiegame.com, ossfind.com — are fully static Astro 5 SSG builds. Articles publish automatically through a GitHub Actions pipeline. The pipeline already handles Dev.to, Hashnode, and Bluesky distribution, plus YouTube thumbnail generation with ffmpeg. I didn't want to add a billed API dependency to this stack.

The options I considered:

Cloudinary with remote transformations: works for on-demand, but requires a paid plan for custom fonts and the transformation URL syntax is brittle to URL-encode correctly.
@vercel/og (Satori-based): excellent for Next.js and Vercel serverless functions, but my sites are static pages on Cloudflare Pages — there's no Edge runtime to serve dynamic OG images from.
node-canvas: full control, zero cost, but native C++ binding compilation in GitHub Actions runners is a recurring pain point. It works, but it adds a non-trivial setup step to CI.
Pillow (Python image library): draws to a bitmap directly. Fine for simple layouts, but anything involving custom fonts, gradients, or CSS flexbox behavior is either impossible or requires dozens of manual coordinate calculations.

The Playwright approach: build an HTML string with CSS, pass it to a headless browser, screenshot it. The browser handles fonts, gradients, flexbox, and every other CSS feature I want to use. No API key. No external service. Just a 160-line Python script and Playwright installed in the runner.

How the HTML template and accent color system works

The script builds a full HTML document as a string, fills in the article title, date, and tags, and hands it to Playwright. The template has a dark card layout with an Inter typeface loaded from Google Fonts CDN.

The one non-obvious piece is the accent color selection. Each article has tags like ["webdev", "astro", "tutorial", "githubactions"]. The script matches these against five regex rules to pick an accent color:

ACCENT_RULES: list[tuple[str, str]] = [
    (r"\b(claude|anthropic|ai|llm|machinelearning)\b", "#8B5CF6"),  # purple
    (r"\b(astro|webdev|tailwindcss|react|nextjs|typescript|javascript)\b", "#0EA5E9"),  # blue
    (r"\b(godot|gamedev|csharp|game|unity)\b", "#22C55E"),  # green
    (r"\b(opensource|github|programming|tutorial)\b", "#F97316"),  # orange
    (r"\b(showdev|indiehackers|productivity)\b", "#F59E0B"),  # amber
]
DEFAULT_ACCENT = "#475569"  # slate fallback

Rules are checked in order; the first match wins. An article tagged ["ai", "webdev"] would pick purple, because ai matches the first rule before webdev matches the second.

The accent color is inserted into the HTML at three points: the background radial gradient (at two different opacity levels: accent + "55" and accent + "33"), the brand mark block, and the tag pill borders. This gives each article a visually distinct color family without requiring any per-article design decision.

Font size also adjusts dynamically: titles over 70 characters render at 54px; shorter titles render at 64px. This is a heuristic that prevents long titles from overflowing the card boundary. It's not perfect for every title, but I haven't needed to manually override anything across 22 articles yet.

The key implementation: `wait_until="networkidle"`

The core Playwright call is:

with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context(viewport={"width": 1200, "height": 630})
    page = context.new_page()

    page.set_content(html, wait_until="networkidle")
    page.screenshot(
        path=str(out_path),
        full_page=False,
        clip={"x": 0, "y": 0, "width": 1200, "height": 630}
    )

The wait_until="networkidle" argument was the critical discovery. Without it, Playwright fires the screenshot as soon as the DOM is ready — before Google Fonts has loaded and applied Inter. The result: the fallback system-ui font renders instead, which looks noticeably different and varies by the runner's OS default.

networkidle tells Playwright to wait until there are no more than 0 network connections for 500ms. In practice this means the Google Fonts CDN request completes and Inter loads before the screenshot fires. This adds roughly 300–500ms per image.

The template includes <link rel="preconnect" href="https://fonts.googleapis.com"> and the corresponding gstatic.com preconnect to minimize the latency. Without preconnect, I saw occasional timeouts where the font didn't load fast enough within the idle window.

The browser instance stays open across all articles

One implementation detail that matters for batch performance: the script opens a single browser instance and reuses the same page object across all articles, calling set_content() in a loop rather than navigating to a URL.

This is faster than opening a new browser per article because Playwright browser startup time is around 500ms. For 22 articles, that's ~11 seconds saved. For 200 articles, it would be ~100 seconds.

The clip parameter on screenshot() is necessary even though the viewport is already set to 1200x630. Without it, Playwright screenshots include a 1px bottom border artifact on some versions of Chromium. The clip forces the exact pixel region I want.

Two image formats from one pipeline

The same GitHub Actions job runs two separate scripts: generate-og.py for the standard 1200×630 OG image (used by Twitter/X, LinkedIn, Dev.to article cards), and generate-summary.py for a 1080×1350 portrait image optimized for Bluesky's visual post format.

The portrait image uses a structured layout with optional sections — card grids, pipeline diagrams, or stat blocks — depending on what summary_data YAML is present in the article frontmatter. Articles that don't have summary_data skip the portrait generation entirely and fall back to the URL card Bluesky generates natively.

This is the same pipeline that runs the post-deploy JSON-LD audit and Bluesky image upload. Adding image generation was a matter of adding two python3 scripts/... steps — no new runner setup beyond pip install playwright && playwright install chromium.

The cover_image auto-patch

One quality-of-life feature: the script writes cover_image: <url> back into the article's frontmatter automatically if it's missing.

def update_cover_image(article_path: Path, slug_base: str) -> bool:
    content = article_path.read_text(encoding="utf-8")
    meta, _ = parse_frontmatter(content)
    if meta.get("cover_image"):
        return False
    cover_url = f"{HOST}/og/articles/{slug_base}.png"
    new_content = re.sub(
        r"^(---\n)([\s\S]*?)(\n---\n)",
        lambda m: m.group(1) + m.group(2) + f"\ncover_image: {cover_url}" + m.group(3),
        content, count=1,
    )
    article_path.write_text(new_content, encoding="utf-8")
    return True

The URL is deterministic — it's the slug plus .png on my CDN. The script generates the image first, then updates the frontmatter. Dev.to and Hashnode both read cover_image from the frontmatter when publishing, so the OG image shows up as the article cover automatically. No manual step.

Comparison: Playwright vs the alternatives

Approach	API cost	CSS control	CI-friendly	On-demand capable	Font flexibility
Playwright + HTML	$0	Full	Yes (slow, ~2s/image)	No	Any web font
Cloudinary transformations	$89/mo at scale	Template only	Yes	Yes	Cloudinary library
@vercel/og (Satori)	$0	JSX subset	Vercel only	Yes	Web fonts via fetch
node-canvas	$0	Full	Needs native build	Yes	System + manual
Pillow + Python	$0	Pixel-level only	Yes	Yes	PIL-loaded fonts

For static sites where every page is a flat HTML file, on-demand OG generation is irrelevant — there's no server to serve it from. Playwright is the only option on this list that gives full CSS control without either native compilation issues or a billed external service.

What I'd change

Bundle Inter locally instead of fetching from Google Fonts. The networkidle approach works, but it means a slow or blocked CDN during CI can cause font loading failures. Bunding the Inter woff2 file in the repo eliminates the network dependency entirely. I haven't done this yet because Google Fonts is convenient and the CDN has been reliable, but a CI failure at 07:00 JST because of a Google CDN blip would motivate the change immediately.

Run images in parallel with asyncio. The synchronous Playwright API processes articles sequentially. For 22 articles at ~2 seconds each, total time is around 45 seconds. For 200 articles, it would be ~7 minutes — too slow for a per-commit CI step. The async Playwright API supports asyncio.gather() for concurrent page instances. I'll need this before the article count gets much larger.

One Playwright instance per image format. Currently generate-og.py and generate-summary.py are separate scripts that each launch their own browser. A single script that generates both formats per article would halve browser launch overhead. Minor at current scale, relevant at 200+ articles.

The hard limit: not suitable for on-demand generation

If you need OG images generated per-request — for a blog where new posts are published dynamically, or for a user-facing tool — this approach doesn't work. Playwright takes 2+ seconds per image and requires a full Chromium binary. Serving that from a request path is impractical.

For on-demand generation at low volume, @vercel/og or a Cloudflare Worker with a canvas API is the right answer. For batch generation at build time in CI, where you control the timing and don't care about per-image latency, Playwright is simpler than any alternative I've found.

FAQ

Q: Why Python instead of Node.js Playwright?

Both Playwright SDKs are functionally equivalent for this use case. I chose Python because my other image-related scripts (generate-summary.py, polish.py) were already Python, and keeping them in one language simplifies the CI setup. The sync_playwright API is slightly more readable than the async Node.js version for sequential batch processing.

Q: Does wait_until="networkidle" always ensure fonts load?

Not guaranteed. networkidle fires when there are zero network connections for 500ms. If the Google Fonts CDN request hasn't started yet when the idle window begins — which can happen if Playwright is very fast at rendering the DOM — the font request comes after the screenshot. In practice, the <link rel="preconnect"> tags I added push the font request early enough that I haven't seen this failure mode. A more reliable approach is to wait for a specific CSS font to be applied using page.wait_for_function("document.fonts.ready").

Q: Can I use this for dynamically generated pages?

Yes, with caveats. You can pass a real URL instead of set_content() using page.goto(url, wait_until="networkidle"). This works well for screenshotting pages that already exist. The timing is less predictable than screenshotting a controlled HTML string because you don't control what JavaScript the page runs.

Q: Why not use Satori directly without @vercel/og?

Satori is an interesting option — it renders JSX to SVG, which you can then convert to PNG with sharp. It's faster per image than Playwright, doesn't require a browser binary, and works in any Node.js environment. The limitation is that it supports a subset of CSS: no background-image: radial-gradient(), no backdrop-filter, limited position support. For my template — which depends on radial gradients for the card background — Satori would require redesigning the layout.

Q: How does the cover image URL work if the PNG isn't published yet?

The cover_image URL points to https://aiappdex.com/og/articles/<slug>.png. The script generates the PNG first and commits it to the repository. Cloudflare Pages deploys the committed file, so by the time the article publishes to Dev.to and Hashnode, the OG image is already live at that URL. The sequence matters: image generation and commit happen before the publish step runs.

Rolling a Google Service Account JWT in Node.js without the googleapis package

MORINAGA — Tue, 09 Jun 2026 03:52:09 +0000

The googleapis npm package is the default answer for calling Google APIs from Node.js. It works, but it installs around 380KB and brings in over 450 transitive dependencies. For a single API used in a CI script — the Search Console URL Inspection API — the underlying auth flow is simple enough to handle directly.

I built scripts/gsc-inspect.mjs to check index status for published URLs. It's about 60 lines, uses three Node.js built-ins (crypto, fetch, URL), and adds zero packages to the repo.

The service account auth flow

Google's service account auth follows RFC 7523 — the JWT Bearer Grant profile of OAuth2. The steps are:

Construct a JWT with your service account's client_email and private key
POST that JWT to https://oauth2.googleapis.com/token
Receive a short-lived access token (valid 3600 seconds)
Use the access token as a Bearer header on API requests

The JWT claims:

const claims = {
  iss: sa.client_email,
  scope: "https://www.googleapis.com/auth/webmasters.readonly",
  aud: "https://oauth2.googleapis.com/token",
  iat: Math.floor(Date.now() / 1000),
  exp: Math.floor(Date.now() / 1000) + 3600,
};

One thing worth knowing upfront: use the webmasters scope, not searchconsole. The URL Inspection API requires webmasters.readonly — the newer searchconsole scope doesn't grant access to it.

Signing with Node's crypto module

Base64url encoding is the only non-obvious part. Standard base64 needs three character replacements to become base64url:

import { createSign } from "node:crypto";

const b64url = (obj) =>
  Buffer.from(JSON.stringify(obj))
    .toString("base64")
    .replace(/=+$/, "")     // strip padding
    .replace(/\+/g, "-")    // + → -
    .replace(/\//g, "_");   // / → _

const unsigned = `${b64url(header)}.${b64url(claims)}`;
const signer = createSign("RSA-SHA256");
signer.update(unsigned);
signer.end();
const sig = signer
  .sign(sa.private_key)
  .toString("base64")
  .replace(/=+$/, "")
  .replace(/\+/g, "-")
  .replace(/\//g, "_");
const jwt = `${unsigned}.${sig}`;

sa.private_key is the RSA private key string from the service account JSON you download from Google Cloud Console. It's already in PKCS#8 PEM format (-----BEGIN PRIVATE KEY-----...), so createSign("RSA-SHA256").sign(key) works directly. No key conversion or external library needed.

Exchanging the JWT for an access token

The token exchange is a URL-encoded form POST:

const res = await fetch("https://oauth2.googleapis.com/token", {
  method: "POST",
  headers: { "content-type": "application/x-www-form-urlencoded" },
  body: new URLSearchParams({
    grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer",
    assertion: jwt,
  }),
});
if (!res.ok) {
  const body = await res.text();
  throw new Error(`Token exchange failed (${res.status}): ${body.slice(0, 300)}`);
}
const { access_token } = await res.json();

The error handling matters here. The invalid_grant error from Google often includes a useful error_description like "Token must expire within 3600 seconds of the issued time" or "Service account not found". Logging the raw response body — truncated to 300 chars — surfaces that directly without digging through a framework's error abstraction.

Calling the URL Inspection endpoint

With the token in hand:

const res = await fetch(
  "https://searchconsole.googleapis.com/v1/urlInspection/index:inspect",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${accessToken}`,
      "content-type": "application/json",
    },
    body: JSON.stringify({
      inspectionUrl: url,
      siteUrl: `https://${siteHost}/`,
    }),
  }
);
const data = await res.json();
console.log(JSON.stringify({
  url,
  coverageState: data.inspectionResult?.indexStatusResult?.coverageState,
  lastCrawlTime: data.inspectionResult?.indexStatusResult?.lastCrawlTime,
}));

The siteUrl field must match a property you've verified in Google Search Console — and it must be the exact string you registered (trailing slash matters). After verifying three domains with Cloudflare DNS TXT records, you also need to add the service account's client_email as a Search Console user (Owner or Full user) before the API will respond to requests.

The coverageState values include INDEXED, SUBMITTED_AND_INDEXED, CRAWLED_CURRENTLY_NOT_INDEXED, and a few others. For post-publish verification, one JSON line per URL is enough — grep-able, loggable in CI without any special tooling.

When not to use this approach

Raw implementation is appropriate for a single API in a CI script. It's less appropriate when:

You're calling multiple Google APIs and want unified auth handling
You need automatic token refresh across long-running processes
You need retry logic, batching, or type-safe API responses
You're shipping production server code where a well-tested library is worth its weight

For this project, the tradeoff is clear: I'm running one CI pipeline across five automated workflows and avoiding unnecessary npm additions. Sixty lines of readable, inspectable code beats 450 transitive dependencies for a use case this narrow.

The implementation is also useful as documentation. The googleapis package abstracts the JWT flow so thoroughly that many developers don't know what's actually happening in the auth exchange. Understanding the raw flow — JWT → token endpoint → Bearer header — makes debugging auth failures faster regardless of what library you end up using.

The sites are still new. I'll publish actual per-URL index coverage data in 30 days once there's something meaningful to report.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

What I learned wiring JSON-LD structured data audits into a post-deploy CI step

MORINAGA — Tue, 09 Jun 2026 03:51:52 +0000

The conclusion first: JSON-LD structured data is one of those things that can vanish from your site without breaking anything visible. The Astro build succeeds. The Cloudflare Pages deploy completes. The page renders fine in a browser. But inside the <script type="application/ld+json"> block — what Googlebot reads to decide whether your page qualifies for rich results — something went wrong, and you won't know until Search Console flags it weeks later.

I added a post-deploy audit step to my CI pipeline that finds this in under 60 seconds. Here's how the script works, what it found on first run, and where the approach falls short.

Why structured data breaks silently in Astro SSG

My three directory sites — aiappdex.com, findindiegame.com, ossfind.com — are fully static Astro 5 SSG builds deployed to Cloudflare Pages. Structured data lives in the <head> of each page, injected by layout components. No server-side rendering, no dynamic injection.

The schema types in use:

SoftwareApplication + BreadcrumbList on aiappdex.com model pages
VideoGame + BreadcrumbList on findindiegame.com game pages
ItemList + BreadcrumbList on ossfind.com alternatives pages
WebSite on all homepages

These all come from Astro layout components. When I add a new slot, reorganize the <head>, or extract shared layout logic, the JSON-LD block can disappear. The Astro compiler doesn't validate structured data. The build step doesn't check it. The deploy succeeds. Nothing errors.

This matters especially for static SSG sites where correctness at build time is the only opportunity — there's no server to validate output at runtime. If a template change drops the VideoGame schema from 2,000 game pages, the damage is done by the time the deploy finishes.

I mentioned in a weekly recap that I suspected some pages had malformed FAQ JSON-LD. That was the nudge to actually build the check.

What the audit script checks

scripts/audit-jsonld.mjs defines a table of expectations per site:

const SITES = [
  {
    host: "aiappdex.com",
    homepage: { path: "/", expectedTypes: ["WebSite"] },
    detail: {
      pathRegex: /\/models\//,
      expectedTypes: ["SoftwareApplication", "BreadcrumbList"],
    },
  },
  {
    host: "findindiegame.com",
    homepage: { path: "/", expectedTypes: ["WebSite"] },
    detail: {
      pathRegex: /\/games\//,
      expectedTypes: ["VideoGame", "BreadcrumbList"],
    },
  },
  {
    host: "ossfind.com",
    homepage: { path: "/", expectedTypes: ["WebSite"] },
    detail: {
      pathRegex: /\/alternatives\//,
      expectedTypes: ["ItemList", "BreadcrumbList"],
    },
  },
];

For each site, the script fetches the homepage and two sample detail pages, extracts all JSON-LD blocks, collects the @type values present, and reports any expected type that's missing.

It runs against live deployed pages, not build output. If Cloudflare returns a cached version of the old page, this catches it. If a CDN edge is serving different HTML than origin, this catches it. Testing the build artifact catches template errors earlier, but not deployment and caching issues — and those are real failure modes I've hit before with Cloudflare Pages.

How it discovers live pages from the sitemap

Instead of hardcoding detail page paths, the script reads the live sitemap to find real pages:

async function discoverDetailPaths(host, regex, count = 2) {
  try {
    const sitemap = await fetch(`https://${host}/sitemap-0.xml`).then(r => r.text());
    const urls = [...sitemap.matchAll(/<loc>([^<]+)<\/loc>/g)].map(m => m[1]);
    return urls.filter(u => regex.test(u)).slice(0, count).map(u => new URL(u).pathname);
  } catch {
    return [];
  }
}

The filename sitemap-0.xml is intentional. As I documented earlier in this series, @astrojs/sitemap on small sites (under roughly 1,000 pages) writes /sitemap-0.xml, not /sitemap-index.xml. Hardcoding /sitemap-index.xml would cause discovery to fail silently — falling back to checking no detail pages at all.

Filtering against pathRegex finds actual model/game/alternative pages that exist in the current production deployment. It checks 2 samples per site per run, which is fast but not exhaustive.

Extracting JSON-LD and handling @graph

The extraction is a regex over the HTML, with one non-obvious case: the @graph unwrapping.

function extractJsonLd(html) {
  const matches = [
    ...html.matchAll(
      /<script[^>]+type=["']application\/ld\+json["'][^>]*>([\s\S]*?)<\/script>/gi,
    ),
  ];
  const items = [];
  for (const m of matches) {
    try {
      const parsed = JSON.parse(m[1].trim());
      const arr = Array.isArray(parsed) ? parsed : [parsed];
      for (const node of arr) {
        if (!node) continue;
        if (node["@graph"]) {
          for (const sub of node["@graph"]) {
            if (sub && sub["@type"]) items.push(sub);
          }
        } else if (node["@type"]) {
          items.push(node);
        }
      }
    } catch (e) {
      items.push({ "@type": "_PARSE_ERROR", error: String(e).slice(0, 80) });
    }
  }
  return items;
}

Some generators bundle multiple schema objects inside a top-level @graph array. Google treats each item in @graph as a separate entity — the audit does the same. If structured data has @graph: [{ "@type": "VideoGame" }, { "@type": "BreadcrumbList" }], both types are extracted and validated individually.

Parse errors surface as a _PARSE_ERROR entry. This catches malformed JSON before it reaches the @type check — useful if a template interpolation injects an unescaped quote into the JSON block.

Adding it to the CI pipeline as a non-fatal step

I wired it into publish-articles.yml — the same pipeline that handles article distribution across Dev.to, Hashnode, and Bluesky:

- name: Audit JSON-LD (non-fatal)
  run: node scripts/audit-jsonld.mjs || echo "JSON-LD audit reported issues (non-fatal)"

The || fallback is the key design decision. It means the step always exits 0, so a failing audit never blocks article publishing. Issues appear in the action log, but no deploy is halted.

This mirrors how I handled the Bluesky image upload timing issue: add the check first, observe what it reports in real conditions, fix the underlying problems, then tighten the failure mode. Making a new check fatal immediately guarantees you'll be debugging a blocked pipeline at the worst moment.

Once all three sites audit clean on every run, I'll drop the || and let a missing BreadcrumbList fail the workflow. Not yet.

What it found on first run

Three issues surfaced immediately:

ossfind.com alternatives pages: missing ItemList

Expected — I hadn't added ItemList schema to the ossfind alternatives layout yet. The audit turned "I should add structured data to ossfind someday" into a concrete, CI-visible task.

findindiegame.com homepage: http:// in WebSite @id

The @id field in the WebSite block was http://findindiegame.com. I had copied a schema template and missed updating the protocol. Nothing breaks visibly — the page renders correctly, the structured data is syntactically valid — but it's inconsistent with what Googlebot sees for the canonical URL.

aiappdex.com model pages: name field used raw HuggingFace model ID

The name field in SoftwareApplication schema contained "meta-llama/Llama-3.1-8B-Instruct" — the raw database ID — instead of the human-readable "Llama 3.1 8B Instruct" that appears in the page <h1>. Both values were available in the Astro component, but the template was pulling from the wrong field.

Site	Issue	Status
ossfind.com	Missing `ItemList` on alternatives pages	Backlog
findindiegame.com	`http://` in WebSite `@id`	Fixed
aiappdex.com	`name` used raw model ID instead of display name	Fixed

Issues 2 and 3 were genuine bugs I wouldn't have found otherwise. Neither showed up in the Astro build, the Cloudflare deploy log, or any browser-level review. The audit found them on first run because it reads structured data the same way Googlebot does: as text inside a <script> tag, not as something the browser renders.

What I'd add next

URL self-consistency check. The http:// bug was caught by manual inspection of the reported types. A systematic check would verify that every url or @id field in structured data matches the actual canonical URL of the page — so that class of error gets caught automatically.

aggregateRating on VideoGame pages. The Steam review data is already in the Turso database: total_reviews, total_positive, review_score. Once I emit aggregateRating structured data, the audit should verify it's present and well-formed on game pages.

FAQPage schema. I want to add FAQ sections to top model pages on aiappdex.com. Once added, the audit needs a validation rule for those pages.

Running against build output before deploy. The current approach finds issues after they're live. Running the same extraction logic against the Astro build output — with a local astro preview server in CI — would catch template regressions pre-deploy. That adds CI complexity I'm not ready to take on; post-deploy detection is good enough for now.

The limit worth stating plainly: the audit checks 2 sample pages per site per run. It doesn't catch issues that only affect specific page types, rare edge cases in the data, or pages that happen not to be in the sitemap sample. It's a smoke test, not a full validation suite.

DEV Community: MORINAGA

How I kept 62 of 80 programmatic pages alive while hiding them from Google

The isCurated gate

Why the gate lives in its own module

Four discovery surfaces gated on the same function

The category layer

What changes automatically

Why I'm abandoning AdSense on two sites and betting on affiliate monetization

The asymmetry between affiliate and AdSense for a zero-traffic site

Why I didn't pivot all three sites

The implementation: monetization mode as env var, not deletion

What affiliate programs I'm actually using

The falsifiable bet

The update I'll actually publish

FAQ

Three sleep intervals for three APIs: Steam 250ms, GitHub 100ms, HuggingFace none

Steam: 250ms, deliberately aggressive

GitHub: 100ms, with authentication doing the real work

HuggingFace: no sleep, because none is needed

A comparison

The pattern that matters

How I built a three-tier content quality ladder for programmatic directory ETL

The three tiers

The upgrade query

The fallback chain

The upsert write

The noindex safety valve

The export step

What I'd do differently

FAQ

Static site search for Astro in 2026: why I picked Pagefind over Algolia and Lunr

The four options I considered

Why Pagefind won

The SearchDialog implementation

What Pagefind doesn't do

Quick comparison table

How I built pairwise AI model compare pages with Claude Haiku and a budget cap

The combinatorics problem

The pair_slug and idempotent inserts

Claude Haiku with system-prompt caching

JSON parsing with a regex fence

Storage in libSQL and the static JSON dump

What I'd change starting over

FAQ

Five overlooked packages running my AI directory stack

tsx — TypeScript without the build ceremony

Pagefind — static full-text search with no server

Crawlee — TypeScript scraping with built-in queue management

eemeli/yaml — small footprint, strict spec compliance

@libsql/client — batched writes are the underrated feature

Five things that caught my attention this week in AI tools and open-source models

1. Claude Code Agent View changes the mental model

2. ZAYA1-8B trained on AMD hardware is a supply chain signal

3. The Harness productivity report has a buried lede

4. ServiceNow Build Agent went GA inside Claude Code and Cursor

5. I removed MCP servers from my pipeline and reliability went up

Turso libSQL vs Cloudflare D1 for an Astro monorepo: the practical difference

The local dev problem D1 doesn't solve cleanly

Why Turso's local fallback changes the calculus

What the migration pattern actually looks like

Where D1 would actually win

Three things I don't know yet

What I learned generating OG images for articles with Playwright and zero API cost

Why I avoided image generation APIs

How the HTML template and accent color system works

The key implementation: wait_until="networkidle"

The browser instance stays open across all articles

Two image formats from one pipeline

The cover_image auto-patch

Comparison: Playwright vs the alternatives

What I'd change

The hard limit: not suitable for on-demand generation

FAQ

Rolling a Google Service Account JWT in Node.js without the googleapis package

The service account auth flow

Signing with Node's crypto module

Exchanging the JWT for an access token

Calling the URL Inspection endpoint

When not to use this approach

What I learned wiring JSON-LD structured data audits into a post-deploy CI step

The key implementation: `wait_until="networkidle"`