DEV Community: MORINAGA

Three post-deploy checks I run after every Cloudflare Pages build

MORINAGA — Thu, 21 May 2026 22:15:55 +0000

After spending two weeks debugging issues that only showed up in production — a sitemap _redirects rule that was blocking my own sitemap-index.xml and a Bluesky image upload race against Cloudflare Pages deploy lag — I added three post-deploy checks to my workflow. They're fast and specific to the failure modes I've actually hit, not a full end-to-end test suite.

Three sites (aiappdex.com, findindiegame.com, ossfind.com) on Cloudflare Pages with Astro 5 SSG. Here's what I check.

Check 1: Sitemap reachability

The simplest check and the one I should have had from day one. After a Cloudflare Pages deploy, I verify that sitemap-index.xml is reachable and returning 200 on all three domains:

for domain in aiappdex.com findindiegame.com ossfind.com; do
  status=$(curl -s -o /dev/null -w "%{http_code}" "https://$domain/sitemap-index.xml")
  echo "$domain/sitemap-index.xml → $status"
  if [ "$status" != "200" ]; then
    echo "FAIL: $domain sitemap unreachable"
  fi
done

I also check sitemap-0.xml — the actual URL sub-sitemap that @astrojs/sitemap generates — and assert that it contains at least a minimum expected URL count. For aiappdex.com that threshold is 1,000; if it drops below that after a deploy, the ETL data pipeline probably broke silently.

The reason this check exists: I had a _redirects rule rewriting sitemap-index.xml → sitemap-0.xml as an emergency workaround that turned out to be wrong. It was live for five days before I found it. The rule was blocking the real sitemap-index.xml from reaching crawlers while appearing fine in the browser (which followed the redirect). Curl with -o /dev/null -w "%{http_code}" doesn't follow redirects by default, so it would have caught this immediately.

Check 2: IndexNow batch submission

After every successful sitemap check, I run node scripts/indexnow.mjs. The script reads the live sitemap XML from each domain, collects all URLs, and POSTs them to the IndexNow endpoint for Bing, Yandex, Naver, and Seznam using site-specific keys.

Output looks like:

aiappdex.com: submitted 1179 URLs → 200 OK
findindiegame.com: submitted 139 URLs → 200 OK
ossfind.com: submitted 144 URLs → 200 OK

If a site returns 403 from IndexNow it usually means the key verification file (/<key>.txt) wasn't deployed correctly or a _redirects rule is mangling the path. Catching this right after deploy matters because the IndexNow key-verification window isn't instantaneous — letting it sit in a broken state delays indexing. I wrote more about the IndexNow setup in this week's tools post.

I run this manually after deploy rather than inline in the GitHub Actions workflow because the Cloudflare Pages build takes 2-3 minutes, and IndexNow works best with live URLs. Running it as a separate workflow_dispatch trigger after the deployment succeeds means I'm submitting URLs that are actually live rather than ones that might still be deploying.

Check 3: Weekly Lighthouse spot-check

The third check runs on a cron — Monday 04:30 UTC — not after every deploy. It's slower (3-4 minutes per site, nine URLs total), so daily would be wasteful for a static site that doesn't change at runtime.

The workflow uses treosh/lighthouse-ci-action with one homepage and one deep entry page per site:

matrix:
  site:
    - { domain: aiappdex.com, sample: /models/timm-vit-base-patch16-clip-224-openai/ }
    - { domain: findindiegame.com, sample: /games/dredge-1562430/ }
    - { domain: ossfind.com, sample: /alternatives/ghost/ }

I'm watching for Performance below 80, CLS above 0.1, or accessibility score regression. Astro SSG with no client-side JS should hold steady on all three — if they slip it means something in Tailwind v4 config or the ad slot component changed the layout paint behavior. The results upload to temporaryPublicStorage so I can diff before/after on regressions.

I don't set hard failure thresholds that block deploys. These sites are pre-revenue with essentially zero traffic right now; blocking a deploy because a Lighthouse score dropped from 94 to 88 would be disproportionate. I treat Lighthouse as a trend monitor, not a gate.

What I'm deliberately not checking

No uptime monitoring — I'm relying on Cloudflare's own infrastructure status. No end-to-end user flow tests. No API availability checks — the Turso DB is only queried at build time in SSG mode, so there's nothing to check at runtime.

For a dynamically rendered site, those gaps would matter. For a static CDN deployment where the entire runtime is pre-built HTML, CSS, and a handful of JSON files, the three checks above cover the actual failure surface I've encountered.

The publish pipeline has its own idempotency layer (it reads published_urls from article frontmatter and skips already-distributed posts), so I don't need to verify cross-posting state after each deploy. That's a separate concern.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Why I'm betting on AI-curated directories when Google AI Overviews answer the same queries

MORINAGA — Thu, 21 May 2026 22:15:52 +0000

The obvious counterargument to everything I'm building is this: Google already does it. You type "best AI tools for video editing" into Google and an AI Overview surfaces a curated list, synthesized from the same kind of data I maintain, without requiring a click. My three directory sites — Top AI Tools, Find Games Like, and Open Alternative To — are competing with a feature baked into the world's dominant search engine.

I launched these sites on 2026-04-23, built on an architecture that runs at about $25/month. Traffic is essentially zero — the sites have been indexed for three weeks and organic crawling takes time. The question I keep returning to isn't whether Google will eventually index my pages. It's whether anyone will prefer clicking through to my site over reading the AI Overview box that already answered the same question.

Here's my honest, falsifiable position.

The bet, stated plainly

By October 2026 — six months post-launch — at least one of the three sites will show organic click trends in Google Search Console indicating real query traffic to specific comparison or filtered-browse pages. I define that as: at least 200 non-homepage organic clicks per month, sustained for two consecutive months, from queries I didn't directly drive through social or newsletter posts.

If that doesn't happen, I'll publish the Search Console screenshots and write a post explaining what I got wrong. I'm committing to that here.

The counterargument I take seriously

AI Overviews have gotten genuinely good at list-and-compare synthesis. If you search "open source alternative to Notion" today, Google often returns a four-item structured list with one-sentence descriptions directly in the Overview box. My Open Alternative To site covers that territory. The AI Overview absorbs the zero-click version of that query.

The optimistic response is: "my site appears as a citation source." The pessimistic response is: "Google consumes your signal and stops sending clicks." The pessimistic version has supporting evidence — industry-wide CTR on informational queries dropped measurably as AI Overviews expanded through 2025, and the trend hasn't reversed.

I don't think the pessimistic version is the whole story, but I'm not dismissing it. The most dangerous move is to assume the counterargument is wrong without designing around it.

Where AI Overviews have structural blind spots

AI Overviews are strong at synthesizing "what exists." They're weaker at three things I've deliberately built for.

Attribute-based filtering. If someone wants "open source Notion alternatives that work offline and have a mobile app," AI Overviews give hedged prose answers because they're synthesizing text, not querying structured fields. My Turso DB has works_offline, has_mobile_app, and last_commit_date as typed columns. Faceted filtering on those fields is something a browseable directory does better than a language model writing a paragraph about the general landscape.

Editorial negative-space. My game recommender includes "avoid if" caveats — structured fields that answer "who should skip this?" generated by a Claude Haiku prompt that specifically forces a critical answer. AI Overviews don't have a mechanism to surface structured negatives. They default to positive framing, which means someone with a specific disqualifying requirement gets an unhelpful answer.

Freshness on maintenance status. The ETL that populates the AI tools directory pulls GitHub commit activity weekly. A tool that hasn't been touched in 14 months is marked as low activity. AI Overviews don't distinguish between a tool actively maintained in 2026 and one that peaked in 2024 — they rely on the recency of web mentions, which can lag by months after a project goes dormant.

None of these defenses are permanent. Google could build structured attribute filtering into AI Overviews. But they require deliberate pipeline design, not just synthesis, and the gap exists now.

The downstream click thesis

Even if my sites lose the zero-click battle on broad discovery terms, there's a second query type I'm explicitly targeting: the downstream comparison query.

The sequence: someone types "Notion alternatives" into Google, gets an AI Overview naming four tools, then types "Appflowy vs Anytype performance" to compare the two they're considering. That second query is post-AI-Overview research. It has commercial intent. It wants a verdict, not another list.

For that query, a page with structured attribute comparison, a clear verdict, and fast load time competes directly with another AI-style answer — and structured data beats generative prose for "which one wins on attribute X." This is partly why I chose static SSG over dynamic AI rendering for these sites: a fast, indexable page with typed comparison fields is what a second-stage research click needs.

Query type	AI Overview strength	Directory strength
Discovery ("best tools for X")	High — often answers directly	Low for zero-click intent
Comparison ("X vs Y, which wins")	Medium — hedges, rarely commits	High — structured attrs + verdict
Filtered browse ("offline + mobile app")	Low — prose, no filters	High — faceted structured data
Freshness ("is X still maintained?")	Inconsistent — lags commits	High — weekly ETL refresh

The comparison and filtered-browse rows are the actual load-bearing columns of this bet.

Why the cost structure matters for intellectual honesty

At $25/month, I can run this experiment for a year without needing revenue to justify continuing. I'm not under pressure to interpret ambiguous signals optimistically.

Compare that to a project burning $200/month on infrastructure: you'd rationalize flat Search Console data as "still in the sandbox phase" past the point where the data actually says something. The full cost breakdown is genuinely minimal — Vercel Pro at $20, Turso starter at $0, Claude Haiku API in single-digit dollars for monthly ETL runs, GitHub Actions on free minutes.

I won't claim AdSense is approved or revenue is flowing until it is. Right now, AdSense rejected the *.vercel.app version of the sites. I've moved to custom domains and verified them in Search Console. I'm waiting for real crawl data before making any claims about what's working.

What would change my mind

Three outcomes would tell me the bet is wrong:

Impressions but near-zero clicks at 90 days. If Search Console shows my pages appearing as AI Overview citation sources but click rates stay near zero on comparison pages specifically, Google is extracting my signal without distributing traffic. That's the worst-case scenario — I'd need to rethink the format entirely.

AdSense keeps rejecting after genuine depth improvements. The original rejection was partly a *.vercel.app domain issue, but if Google's classifier still rates the pages as thin after I've rebuilt with real structured content and specific editorial attributes, my model of what "quality" means to the classifier is wrong.

Comparison queries migrate fully to LLM chat. If people stop typing "X vs Y" into Google and start asking ChatGPT directly, the downstream click I'm betting on disappears. I don't see evidence of this happening at scale for research involving specific attribute constraints — but I'm monitoring query volume patterns month-over-month.

The first outcome is the one I'd want to see early. Impressions with near-zero clicks on comparison pages by month 3 would tell me to pivot the format immediately rather than wait six months for a conclusion I could have reached sooner.

FAQ

Why three sites instead of one authority site?

Three narrow sites let me test three different intent types simultaneously. Games-like, AI tools, and OSS alternatives attract different queries and different audiences. One site would take longer to produce the same signal volume about which format works. The original architecture post covers the reasoning.

How does Claude Haiku generate the structured editorial fields?

Each ETL run sends entries through a shared Claude Haiku client that uses system-prompt caching to amortize the cost across batch runs. The prompts are tuned to force specific attribute outputs — avoid-if caveats, audience fit, freshness status — not open-ended descriptions.

What if one site works and two don't?

That's a useful outcome, not a failure. The format that works tells me something specific about the intent type. I'll invest in what works and document what didn't.

Where will you publish the October 2026 verdict?

On this blog, with raw Search Console screenshots. I'll publish regardless of whether the numbers are favorable.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Static site search for Astro in 2026: why I picked Pagefind over Algolia and Lunr

MORINAGA — Wed, 20 May 2026 22:12:46 +0000

I added search to all three of my AI-curated directory sites last month. The choice wasn't obvious — there are at least four options with real adoption — so here's the breakdown I actually ran through before landing on Pagefind.

The four options I considered

Pagefind is a Rust-based static search library. It runs at build time, generates an index in /_pagefind/, and serves everything as static files. No backend, no API key, no per-query billing. It ships a prebuilt UI (PagefindUI) that you can mount on any element, and it supports WebAssembly for in-browser querying.

Algolia DocSearch is free for open-source documentation sites, $49/month for commercial sites below a certain crawl limit. It indexes your content via their crawler (or an API push), stores it on Algolia's infrastructure, and gives you a hosted search widget. Fast, polished, and battle-tested — it's what most major docs sites use.

Lunr.js is a client-side search library. You build the index at build time, serialize it to JSON, and ship it with the page. The browser loads the entire index on first search. Works offline, no external dependency, but the index size grows linearly with content, and there's no incremental loading.

FlexSearch is a newer alternative to Lunr with better performance characteristics and smaller bundle size, but the same core trade-off: you ship the whole index to the browser upfront.

Why Pagefind won

The decisive factor was index size management. My directories have 500-1,000 entries per site, each with a multi-paragraph generated description. A Lunr index for 1,000 entries would be 2-4MB shipped with every page load. Pagefind shards its index and loads chunks lazily as the user types — so the initial load is under 30KB (the WASM binary + a small manifest), and individual chunk fetches happen on demand.

The second factor was cost. Algolia DocSearch's commercial tier runs $49/month per site. I'm running three sites on a total infrastructure budget of roughly $25/month. Pagefind is free.

The third factor was the deploy model. Because everything in /_pagefind/ is a static file, Cloudflare Pages caches it at the edge with no configuration. There's no API to rate-limit, no service availability to depend on, no API key to rotate.

The SearchDialog implementation

The search component is a <dialog> element with a Pagefind UI mounted inside it. I load the pagefind-ui.js script lazily — only when the dialog is first opened — to keep it off the critical path:

function loadPagefind() {
  if (loaded || !root) return;
  loaded = true;
  var s = document.createElement("script");
  s.src = "/_pagefind/pagefind-ui.js";
  s.onload = function () {
    if (window.PagefindUI) {
      new window.PagefindUI({ element: root, showSubResults: true, resetStyles: false });
    }
  };
  s.onerror = function () {
    root.innerHTML = '<p>Search index not available yet (first build). Try again after next deploy.</p>';
  };
  document.head.appendChild(s);
}

The s.onerror handler is the part most tutorials skip. On the first deploy of a new Cloudflare Pages site, the /_pagefind/ directory doesn't exist yet — Pagefind only runs during the build. If a user opens search before the first full build completes, pagefind-ui.js 404s. Without the error handler, you get a silent failure. With it, you get a legible message.

The <dialog> element is the right primitive here: it handles focus trapping automatically, Escape closes it natively, and backdrop: CSS pseudo-element gives you the dimmed overlay without JavaScript. The Cmd+K keyboard shortcut is wired with document.addEventListener("keydown", ...) — no library needed.

What Pagefind doesn't do

Two gaps I've hit:

No query logging. Pagefind runs entirely in the browser and doesn't send queries anywhere. For a commercial directory, knowing what users search for is valuable — it tells you which models or games to add, and which compare pages to prioritize. With Algolia you get this for free. With Pagefind you'd need to add a thin logging layer (a fetch POST to an analytics endpoint on each query event). I haven't built this yet.

No fuzzy matching out of the box. Pagefind does stemming and basic substring matching, but "stabilty diffusion" (typo) won't match "stable diffusion". Algolia's typo-tolerance is significantly better. For an AI tools directory where model names are long and often misremembered, this matters. I'll probably add a query-suggestion layer that does fuzzy pre-matching before handing off to Pagefind.

Quick comparison table

	Pagefind	Algolia DocSearch	Lunr.js
Cost	Free	$49/mo (commercial)	Free
Index location	Static files	Algolia cloud	Shipped with page
Initial JS load	~30KB	~80KB	~10KB + index
Index size scalability	Chunked, lazy	Server-side	Linear, upfront
Typo tolerance	Basic stemming	Strong	Weak
Query logging	No	Yes	No
Build-time integration	Yes	Crawler / push API	Yes

For a static site on a tight infrastructure budget with 500-1,000 entries, Pagefind is the right default. If the site were larger or if I needed typo tolerance and query analytics without building them myself, Algolia would be worth the cost.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

How I built pairwise AI model compare pages with Claude Haiku and a budget cap

MORINAGA — Wed, 20 May 2026 22:12:43 +0000

When I added compare pages to the Top AI Tools directory, the first question I had to answer was: how many pairs am I actually looking at? With roughly 200 models across 8 pipeline tags, the naive upper bound is 200 × 199 / 2 ≈ 19,900 pairs. Generating content for each one with Claude Haiku would cost somewhere around $20 per run — not ruinous, but not something I wanted to run daily without thinking carefully.

Here's what I actually built, where it falls short, and what I'd do differently if starting over.

The combinatorics problem

Model compare pages exist for a specific type of query: "llama 3 vs mistral 7b", "stable diffusion vs sdxl", "whisper vs wav2vec2". These are high-intent queries — the user has already narrowed down to a shortlist and wants a concrete decision nudge. The static SSG approach I'm running means I need to precompute each compare page at build time, which puts pressure on how many pages I can afford to generate.

The solution I landed on: group by pipeline_tag, pair the top-4 models by download count within each group, then cap total pairs with a COMPARE_LIMIT env var. Within a single pipeline like text-generation, the top 4 models give 6 pairs (4 choose 2). Across 8 active pipelines that's roughly 48 pairs. The env cap of 50 means I stay within that budget while having room to grow.

const byPipe = new Map<string, typeof models>();
for (const m of models) {
  if (!m.pipeline_tag) continue;
  const arr = byPipe.get(m.pipeline_tag) ?? [];
  arr.push(m);
  byPipe.set(m.pipeline_tag, arr);
}

const pairs: Array<[Model, Model]> = [];
for (const [, list] of byPipe) {
  const sorted = [...list].sort((a, b) => b.downloads - a.downloads);
  const take = sorted.slice(0, Math.min(4, sorted.length));
  for (let i = 0; i < take.length; i++) {
    for (let j = i + 1; j < take.length; j++) {
      pairs.push([take[i]!, take[j]!]);
    }
  }
}
const chosen = pairs.slice(0, MAX);

The pairing happens entirely within pipelines right now, which means I'm covering "llama vs mistral" (both text-generation) but not "whisper vs gemini-vision" (cross-pipeline). Cross-pipeline comparisons are actually more valuable for users who don't know the landscape yet — that's the next iteration.

The pair_slug and idempotent inserts

The slug for each compare pair is constructed deterministically: sort the two model slugs alphabetically, join with --vs--. So whether the ETL processes (llama-3, mistral-7b) or (mistral-7b, llama-3), the slug is always llama-3--vs--mistral-7b.

const pairSlug = [a.slug, b.slug].sort().join("--vs--");

This makes the entire ETL idempotent. The script runs every night. If all pairs already exist in the DB, it exits in a couple of seconds without a single Claude call. I check before inserting rather than using INSERT OR IGNORE at the SQL level — the explicit check lets me count skipped vs generated in the same run, which I log:

[compare] done — generated: 3, skipped: 47

This matters for monitoring. A run that generates 0 and skips 50 is healthy. A run that generates 0 and skips 0 (nothing in DB, nothing processed) would indicate a bug.

Claude Haiku with system-prompt caching

I reuse the shared Haiku client I built in week one, which handles cacheSystem: true on the system prompt. Since the system prompt — the JSON schema instruction — is identical across all compare calls, the first call primes the cache and subsequent calls see near-zero token cost on that prefix.

The user prompt includes both model names, their authors, pipeline tags, and up to 400 characters of their existing summaries (which come from the earlier content generation step):

const userPrompt = `Compare these two AI models:
A: ${a.name} (author: ${a.author ?? "unknown"}, pipeline: ${a.pipeline_tag ?? "unknown"})
   Summary: ${a.summary?.slice(0, 400) ?? "(none)"}
B: ${b.name} (author: ${b.author ?? "unknown"}, pipeline: ${b.pipeline_tag ?? "unknown"})
   Summary: ${b.summary?.slice(0, 400) ?? "(none)"}

Produce the JSON comparison.`;

Truncating summaries at 400 characters keeps the user prompt lean. Compare pages are about the delta between two models, not a rehash of each model individually. I already have dedicated model pages for depth; the compare page needs to answer "which one, for what" — that takes maybe 6 sentences total.

The system prompt requests a JSON object with summary, differences (array), similarities (array), and recommendation. Keeping the output shape narrow means Haiku rarely wanders off-schema.

JSON parsing with a regex fence

Even with tight prompting, Haiku occasionally produces JSON with an explanation preamble: "Here is the comparison:" followed by the actual object. Strict JSON.parse on the raw output would throw. I extract the outermost {...} block with a regex before parsing:

function parseCompare(text: string, fb: CompareData): CompareData {
  try {
    const m = text.match(/\{[\s\S]*\}/);
    if (!m) return fb;
    const p = JSON.parse(m[0]);
    return {
      summary: typeof p.summary === "string" ? p.summary : fb.summary,
      differences: Array.isArray(p.differences)
        ? p.differences.map(String)
        : fb.differences,
      similarities: Array.isArray(p.similarities)
        ? p.similarities.map(String)
        : fb.similarities,
      recommendation:
        typeof p.recommendation === "string"
          ? p.recommendation
          : fb.recommendation,
    };
  } catch {
    return fb;
  }
}

Each field is validated individually before being accepted. If differences comes back as a string (occasional Haiku behavior when it conflates the array with a comma-separated list), the page falls back to the template for that field rather than crashing.

The fallback struct is worth writing carefully. I spent five minutes on mine and it shows:

const fb: CompareData = {
  summary: `${a.name} and ${b.name} are both ${a.pipeline_tag} models. See each entry for specifics.`,
  differences: ["See individual model pages for architecture and use cases."],
  similarities: ["Both are open-source models on HuggingFace."],
  recommendation: "Pick based on your compute budget and specific task requirements.",
};

A user landing on a fallback-generated compare page gets a technically-true page that directs them to the model pages rather than a blank or error state. The model_used column in the DB records "fallback-template" for these rows, which I use to identify candidates for regeneration.

Storage in libSQL and the static JSON dump

Compare data lives in a model_compare table in Turso libSQL, with a unique constraint on pair_slug. After the ETL loop, everything gets dumped to compare.json for the static build:

const all = await db.execute(
  `SELECT * FROM model_compare ORDER BY slug_a, slug_b`
);
const entries = all.rows.map((r) => ({
  slug_a: String(r.slug_a),
  slug_b: String(r.slug_b),
  pair_slug: String(r.pair_slug),
  summary: r.summary ? String(r.summary) : "",
  differences: r.differences ? JSON.parse(String(r.differences)) as string[] : [],
  similarities: r.similarities ? JSON.parse(String(r.similarities)) as string[] : [],
  recommendation: r.recommendation ? String(r.recommendation) : "",
}));
await writeFile("./src/data/compare.json", JSON.stringify(entries, null, 2));

The Astro build reads this JSON at build time, generating one static page per pair. No runtime DB calls, no cold starts. The tradeoff is freshness: compare content is up to 24 hours stale. For "llama 3.1 vs llama 3.2", that's fine — the models don't change daily.

I validate the JSON-LD on compare pages through the post-deploy audit CI step the same way I do for individual model pages. Structured data matters more on comparison queries because those are the exact queries that AI Overviews tend to surface, so getting the schema right is worth the CI overhead.

The Astro slug generation for compare pages uses the pair_slug directly. The URL pattern is /compare/llama-3--vs--mistral-7b/, which is ugly but unambiguous — the double-dash separator makes it clear this is a two-part slug rather than a hyphen in a model name.

What I'd change starting over

Generate cross-pipeline pairs from day one. The most useful compare queries aren't "llama 3.1 vs llama 3.2" — users who care about that distinction already know. The interesting queries are cross-category: "should I run inference on a text-generation model or use a RAG pipeline?" I skipped this to stay within the budget cap, but it means I'm missing the long-tail traffic that would actually be differentiated from generic model pages.

Drive pair selection from search query logs. Right now I pick pairs by download rank. A better signal would be which pairs users actually search for. Pagefind runs client-side and doesn't log queries to any server, so I'd need a thin logging endpoint — something like a POST to a GitHub Actions-triggered function that appends to a JSONL file. Then the ETL reads the top-N ungenerated pairs from the log. This is a small amount of infrastructure but it would make the pair selection much more demand-driven.

Raise the budget cap. MAX=50 is conservative. At current Haiku pricing with prompt caching, 500 pairs would cost roughly $0.10 per nightly run. I was cautious when I set the default, but I've watched the billing closely and the actual spend is a fraction of what I modeled. I'll bump this to 200 in the next ETL config update.

The itch.io entries pattern I added to the indie-games directory taught me to plan for the second data source earlier. Compare pages have the same shape: a join between two rows. Getting the abstraction right before you have 500+ rows in the DB is much easier than retrofitting it.

FAQ

Does the ETL run every night even when no new models are added?

Yes, but it's nearly free when nothing is new. The check-before-insert means most nights it does 50 DB reads and exits in under 3 seconds without touching the Claude API. The console output shows generated: 0, skipped: 47 which is the signal that everything is up to date.

What happens when Claude returns malformed JSON?

parseCompare catches the error and returns the fallback struct. The row is still written to the DB with model_used = "fallback-template", which I can query to find rows worth retrying. In practice, this happens on maybe 2-3% of generations — usually when the two models have very sparse metadata and Haiku doesn't have enough context to produce structured output.

Does the compare.json file get unwieldy as pairs accumulate?

At 50 pairs it's roughly 25KB. At 500 pairs it'll be around 250KB — still fine for build-time loading in Astro. If I ever hit 5,000 pairs I'd split the file by pipeline_tag and lazy-import only the relevant subset for each page. For now, one flat JSON file is simpler and fast enough.

Why not compute compare content at request time with an edge function?

Cold starts and cost. An edge function hit for each compare page view would add 200-500ms of latency (Haiku inference + DB round trip) and would cost much more per-pageview than the nightly batch approach. The content also doesn't need to be fresher than daily — model capabilities don't shift on an hourly basis. Static precomputation is the right tradeoff here, consistent with the broader bet on static SSG I'm running on all three sites.

How do you handle the case where a model is removed from HuggingFace?

Right now, I don't. If model foo is deleted from HuggingFace but its compare rows are still in the DB, those compare pages will still be served at build time. They'll have the old data until the model's row in models.json is removed — which only happens if the model falls out of the top-500 in the nightly fetch. It's a known gap. For now, the risk is low; popular models don't disappear. A more robust system would cross-reference the compare table against the model table and tombstone orphaned pairs.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Three post-deploy checks I run after every Cloudflare Pages build

MORINAGA — Wed, 20 May 2026 22:12:08 +0000

Three sites (aiappdex.com, findindiegame.com, ossfind.com) on Cloudflare Pages with Astro 5 SSG. Here's what I check.

Check 1: Sitemap reachability

The simplest check and the one I should have had from day one. After a Cloudflare Pages deploy, I verify that sitemap-index.xml is reachable and returning 200 on all three domains:

for domain in aiappdex.com findindiegame.com ossfind.com; do
  status=$(curl -s -o /dev/null -w "%{http_code}" "https://$domain/sitemap-index.xml")
  echo "$domain/sitemap-index.xml → $status"
  if [ "$status" != "200" ]; then
    echo "FAIL: $domain sitemap unreachable"
  fi
done

Check 2: IndexNow batch submission

Output looks like:

aiappdex.com: submitted 1179 URLs → 200 OK
findindiegame.com: submitted 139 URLs → 200 OK
ossfind.com: submitted 144 URLs → 200 OK

Check 3: Weekly Lighthouse spot-check

The workflow uses treosh/lighthouse-ci-action with one homepage and one deep entry page per site:

matrix:
  site:
    - { domain: aiappdex.com, sample: /models/timm-vit-base-patch16-clip-224-openai/ }
    - { domain: findindiegame.com, sample: /games/dredge-1562430/ }
    - { domain: ossfind.com, sample: /alternatives/ghost/ }

What I'm deliberately not checking

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Why I'm betting on AI-curated directories when Google AI Overviews answer the same queries

MORINAGA — Wed, 20 May 2026 22:11:59 +0000

Here's my honest, falsifiable position.

The bet, stated plainly

If that doesn't happen, I'll publish the Search Console screenshots and write a post explaining what I got wrong. I'm committing to that here.

The counterargument I take seriously

I don't think the pessimistic version is the whole story, but I'm not dismissing it. The most dangerous move is to assume the counterargument is wrong without designing around it.

Where AI Overviews have structural blind spots

AI Overviews are strong at synthesizing "what exists." They're weaker at three things I've deliberately built for.

None of these defenses are permanent. Google could build structured attribute filtering into AI Overviews. But they require deliberate pipeline design, not just synthesis, and the gap exists now.

The downstream click thesis

Even if my sites lose the zero-click battle on broad discovery terms, there's a second query type I'm explicitly targeting: the downstream comparison query.

Query type	AI Overview strength	Directory strength
Discovery ("best tools for X")	High — often answers directly	Low for zero-click intent
Comparison ("X vs Y, which wins")	Medium — hedges, rarely commits	High — structured attrs + verdict
Filtered browse ("offline + mobile app")	Low — prose, no filters	High — faceted structured data
Freshness ("is X still maintained?")	Inconsistent — lags commits	High — weekly ETL refresh

The comparison and filtered-browse rows are the actual load-bearing columns of this bet.

Why the cost structure matters for intellectual honesty

At $25/month, I can run this experiment for a year without needing revenue to justify continuing. I'm not under pressure to interpret ambiguous signals optimistically.

What would change my mind

Three outcomes would tell me the bet is wrong:

FAQ

Why three sites instead of one authority site?

How does Claude Haiku generate the structured editorial fields?

What if one site works and two don't?

That's a useful outcome, not a failure. The format that works tells me something specific about the intent type. I'll invest in what works and document what didn't.

Where will you publish the October 2026 verdict?

On this blog, with raw Search Console screenshots. I'll publish regardless of whether the numbers are favorable.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Three post-deploy checks I run after every Cloudflare Pages build

MORINAGA — Tue, 19 May 2026 22:12:50 +0000

Three sites (aiappdex.com, findindiegame.com, ossfind.com) on Cloudflare Pages with Astro 5 SSG. Here's what I check.

Check 1: Sitemap reachability

The simplest check and the one I should have had from day one. After a Cloudflare Pages deploy, I verify that sitemap-index.xml is reachable and returning 200 on all three domains:

for domain in aiappdex.com findindiegame.com ossfind.com; do
  status=$(curl -s -o /dev/null -w "%{http_code}" "https://$domain/sitemap-index.xml")
  echo "$domain/sitemap-index.xml → $status"
  if [ "$status" != "200" ]; then
    echo "FAIL: $domain sitemap unreachable"
  fi
done

Check 2: IndexNow batch submission

Output looks like:

aiappdex.com: submitted 1179 URLs → 200 OK
findindiegame.com: submitted 139 URLs → 200 OK
ossfind.com: submitted 144 URLs → 200 OK

Check 3: Weekly Lighthouse spot-check

The workflow uses treosh/lighthouse-ci-action with one homepage and one deep entry page per site:

matrix:
  site:
    - { domain: aiappdex.com, sample: /models/timm-vit-base-patch16-clip-224-openai/ }
    - { domain: findindiegame.com, sample: /games/dredge-1562430/ }
    - { domain: ossfind.com, sample: /alternatives/ghost/ }

What I'm deliberately not checking

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Why I'm betting on AI-curated directories when Google AI Overviews answer the same queries

MORINAGA — Tue, 19 May 2026 22:12:48 +0000

Here's my honest, falsifiable position.

The bet, stated plainly

If that doesn't happen, I'll publish the Search Console screenshots and write a post explaining what I got wrong. I'm committing to that here.

The counterargument I take seriously

I don't think the pessimistic version is the whole story, but I'm not dismissing it. The most dangerous move is to assume the counterargument is wrong without designing around it.

Where AI Overviews have structural blind spots

AI Overviews are strong at synthesizing "what exists." They're weaker at three things I've deliberately built for.

None of these defenses are permanent. Google could build structured attribute filtering into AI Overviews. But they require deliberate pipeline design, not just synthesis, and the gap exists now.

The downstream click thesis

Even if my sites lose the zero-click battle on broad discovery terms, there's a second query type I'm explicitly targeting: the downstream comparison query.

Query type	AI Overview strength	Directory strength
Discovery ("best tools for X")	High — often answers directly	Low for zero-click intent
Comparison ("X vs Y, which wins")	Medium — hedges, rarely commits	High — structured attrs + verdict
Filtered browse ("offline + mobile app")	Low — prose, no filters	High — faceted structured data
Freshness ("is X still maintained?")	Inconsistent — lags commits	High — weekly ETL refresh

The comparison and filtered-browse rows are the actual load-bearing columns of this bet.

Why the cost structure matters for intellectual honesty

At $25/month, I can run this experiment for a year without needing revenue to justify continuing. I'm not under pressure to interpret ambiguous signals optimistically.

What would change my mind

Three outcomes would tell me the bet is wrong:

FAQ

Why three sites instead of one authority site?

How does Claude Haiku generate the structured editorial fields?

What if one site works and two don't?

That's a useful outcome, not a failure. The format that works tells me something specific about the intent type. I'll invest in what works and document what didn't.

Where will you publish the October 2026 verdict?

On this blog, with raw Search Console screenshots. I'll publish regardless of whether the numbers are favorable.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Three post-deploy checks I run after every Cloudflare Pages build

MORINAGA — Mon, 18 May 2026 22:12:21 +0000

Three sites (aiappdex.com, findindiegame.com, ossfind.com) on Cloudflare Pages with Astro 5 SSG. Here's what I check.

Check 1: Sitemap reachability

The simplest check and the one I should have had from day one. After a Cloudflare Pages deploy, I verify that sitemap-index.xml is reachable and returning 200 on all three domains:

for domain in aiappdex.com findindiegame.com ossfind.com; do
  status=$(curl -s -o /dev/null -w "%{http_code}" "https://$domain/sitemap-index.xml")
  echo "$domain/sitemap-index.xml → $status"
  if [ "$status" != "200" ]; then
    echo "FAIL: $domain sitemap unreachable"
  fi
done

Check 2: IndexNow batch submission

Output looks like:

aiappdex.com: submitted 1179 URLs → 200 OK
findindiegame.com: submitted 139 URLs → 200 OK
ossfind.com: submitted 144 URLs → 200 OK

Check 3: Weekly Lighthouse spot-check

The workflow uses treosh/lighthouse-ci-action with one homepage and one deep entry page per site:

matrix:
  site:
    - { domain: aiappdex.com, sample: /models/timm-vit-base-patch16-clip-224-openai/ }
    - { domain: findindiegame.com, sample: /games/dredge-1562430/ }
    - { domain: ossfind.com, sample: /alternatives/ghost/ }

What I'm deliberately not checking

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Why I'm betting on AI-curated directories when Google AI Overviews answer the same queries

MORINAGA — Mon, 18 May 2026 22:12:19 +0000

Here's my honest, falsifiable position.

The bet, stated plainly

If that doesn't happen, I'll publish the Search Console screenshots and write a post explaining what I got wrong. I'm committing to that here.

The counterargument I take seriously

I don't think the pessimistic version is the whole story, but I'm not dismissing it. The most dangerous move is to assume the counterargument is wrong without designing around it.

Where AI Overviews have structural blind spots

AI Overviews are strong at synthesizing "what exists." They're weaker at three things I've deliberately built for.

None of these defenses are permanent. Google could build structured attribute filtering into AI Overviews. But they require deliberate pipeline design, not just synthesis, and the gap exists now.

The downstream click thesis

Even if my sites lose the zero-click battle on broad discovery terms, there's a second query type I'm explicitly targeting: the downstream comparison query.

Query type	AI Overview strength	Directory strength
Discovery ("best tools for X")	High — often answers directly	Low for zero-click intent
Comparison ("X vs Y, which wins")	Medium — hedges, rarely commits	High — structured attrs + verdict
Filtered browse ("offline + mobile app")	Low — prose, no filters	High — faceted structured data
Freshness ("is X still maintained?")	Inconsistent — lags commits	High — weekly ETL refresh

The comparison and filtered-browse rows are the actual load-bearing columns of this bet.

Why the cost structure matters for intellectual honesty

At $25/month, I can run this experiment for a year without needing revenue to justify continuing. I'm not under pressure to interpret ambiguous signals optimistically.

What would change my mind

Three outcomes would tell me the bet is wrong:

FAQ

Why three sites instead of one authority site?

How does Claude Haiku generate the structured editorial fields?

What if one site works and two don't?

That's a useful outcome, not a failure. The format that works tells me something specific about the intent type. I'll invest in what works and document what didn't.

Where will you publish the October 2026 verdict?

On this blog, with raw Search Console screenshots. I'll publish regardless of whether the numbers are favorable.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Three post-deploy checks I run after every Cloudflare Pages build

MORINAGA — Sun, 17 May 2026 22:12:32 +0000

Three sites (aiappdex.com, findindiegame.com, ossfind.com) on Cloudflare Pages with Astro 5 SSG. Here's what I check.

Check 1: Sitemap reachability

The simplest check and the one I should have had from day one. After a Cloudflare Pages deploy, I verify that sitemap-index.xml is reachable and returning 200 on all three domains:

for domain in aiappdex.com findindiegame.com ossfind.com; do
  status=$(curl -s -o /dev/null -w "%{http_code}" "https://$domain/sitemap-index.xml")
  echo "$domain/sitemap-index.xml → $status"
  if [ "$status" != "200" ]; then
    echo "FAIL: $domain sitemap unreachable"
  fi
done

Check 2: IndexNow batch submission

Output looks like:

aiappdex.com: submitted 1179 URLs → 200 OK
findindiegame.com: submitted 139 URLs → 200 OK
ossfind.com: submitted 144 URLs → 200 OK

Check 3: Weekly Lighthouse spot-check

The workflow uses treosh/lighthouse-ci-action with one homepage and one deep entry page per site:

matrix:
  site:
    - { domain: aiappdex.com, sample: /models/timm-vit-base-patch16-clip-224-openai/ }
    - { domain: findindiegame.com, sample: /games/dredge-1562430/ }
    - { domain: ossfind.com, sample: /alternatives/ghost/ }

What I'm deliberately not checking

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Why I'm betting on AI-curated directories when Google AI Overviews answer the same queries

MORINAGA — Sun, 17 May 2026 22:12:28 +0000

Here's my honest, falsifiable position.

The bet, stated plainly

If that doesn't happen, I'll publish the Search Console screenshots and write a post explaining what I got wrong. I'm committing to that here.

The counterargument I take seriously

I don't think the pessimistic version is the whole story, but I'm not dismissing it. The most dangerous move is to assume the counterargument is wrong without designing around it.

Where AI Overviews have structural blind spots

AI Overviews are strong at synthesizing "what exists." They're weaker at three things I've deliberately built for.

None of these defenses are permanent. Google could build structured attribute filtering into AI Overviews. But they require deliberate pipeline design, not just synthesis, and the gap exists now.

The downstream click thesis

Even if my sites lose the zero-click battle on broad discovery terms, there's a second query type I'm explicitly targeting: the downstream comparison query.

Query type	AI Overview strength	Directory strength
Discovery ("best tools for X")	High — often answers directly	Low for zero-click intent
Comparison ("X vs Y, which wins")	Medium — hedges, rarely commits	High — structured attrs + verdict
Filtered browse ("offline + mobile app")	Low — prose, no filters	High — faceted structured data
Freshness ("is X still maintained?")	Inconsistent — lags commits	High — weekly ETL refresh

The comparison and filtered-browse rows are the actual load-bearing columns of this bet.

Why the cost structure matters for intellectual honesty

At $25/month, I can run this experiment for a year without needing revenue to justify continuing. I'm not under pressure to interpret ambiguous signals optimistically.

What would change my mind

Three outcomes would tell me the bet is wrong:

FAQ

Why three sites instead of one authority site?

How does Claude Haiku generate the structured editorial fields?

What if one site works and two don't?

That's a useful outcome, not a failure. The format that works tells me something specific about the intent type. I'll invest in what works and document what didn't.

Where will you publish the October 2026 verdict?

On this blog, with raw Search Console screenshots. I'll publish regardless of whether the numbers are favorable.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.