Week 2 of running three AI-curated directory sites. Five things that caught me off-guard between Monday and Friday.
1. My sitemap 404 was not @astrojs/sitemap skipping the index file
I spent most of Tuesday assuming Google Search Console was silently ignoring my sitemap submission. It wasn't — and my next guess was wrong too. A smoke check returned 404 on /sitemap-index.xml, so I concluded that @astrojs/sitemap only emits /sitemap-0.xml when a site's URLs fit into a single file. It doesn't work that way: local builds of all three sites produce sitemap-0.xml and sitemap-index.xml, including the two sites with only 139 and 144 URLs.
What I actually shipped on the day was a Cloudflare _redirects rewrite (/sitemap-index.xml → /sitemap-0.xml), plus a re-submit in GSC and an IndexNow re-ping. Treat that rewrite as a stopgap built on a diagnosis that didn't hold — the plugin was generating the index file the whole time, so the rewrite was masking it rather than fixing anything.
2. Cloudflare Pages returns HTTP 500 on pages with certain URL substrings
One of my sites stores open-source alternative listings that include DigitalOcean referral links. Every page containing the string m.do.co threw a silent HTTP 500 from Cloudflare's edge — no error logs, no detail, just 500. Local dev preview and Vercel were both fine.
The workaround was encoding the problematic substring in stored content before it reaches the HTML output. Isolating it took longer than it should have because the behavior only triggers on-edge. All three sites are still on Cloudflare Pages — encoding the substring was cheaper than moving hosts again a week after the migration.
3. IndexNow + Wayback Machine is a free GitHub Actions addition
After the sitemap gap, I wired two things into GitHub Actions:
- An IndexNow ping at the end of the article-publish workflow —
node scripts/indexnow.mjswalks each site's sitemap and submits the URLs in bulk (covers Bing, Yandex, Naver, Seznam) - A separate monthly cron workflow that POSTs each site's home page and
sitemap.xmlto the Wayback Machine public save API
The Wayback job creates a timestamped public record of when the sites existed and what they listed — my main motivation is site-age evidence for sale documentation later. Whether either one influences Google's crawl schedule I genuinely don't know yet — I'll publish crawl data at the 30-day mark. Both are free: the IndexNow ping rides along with each article publish, the Wayback job runs on its own monthly schedule.
4. You can add a second content source to a single-source directory without schema forks
My indie game directory was built entirely around the Steam API: appId as primary key, all metadata derived from Steam's schema. Adding itch.io games meant introducing a source that has no equivalent of appId.
The solution was two optional fields on the shared GameEntry type: source (steam | itch) and external_url, which is derived from appid at render time for Steam entries and stored directly for itch.io ones. The itch entries live in a hand-maintained curated.json that gets merged with the Steam export at build time, so the Steam ETL and its schema didn't change at all. Week 2 isn't the right moment to build a multi-source abstraction layer — one entry type with two optional paths works until it stops working, and that inflection point is later than you think.
5. Hashnode sets itself as canonical if you post there without originalArticleURL
When cross-posting to Dev.to and Hashnode, order and explicit fields matter. Post to Hashnode first — or post without providing originalArticleURL — and Hashnode treats its own URL as the canonical. That means Google may index the Hashnode version as authoritative, not ideal if Dev.to is your intended primary channel.
Fix: in packages/publish/src/hashnode.ts, fall back to the Dev.to URL already recorded in the article's published_urls frontmatter and pass that as originalArticleURL in the Hashnode GraphQL mutation. One extra branch, three lines. Dev.to publishes first in the same run, so the URL was already sitting in the pipeline — it just wasn't being passed downstream.
Three more weeks before GSC data becomes meaningful. Next week I want to look at structured data validation — I suspect a few pages have malformed FAQ JSON-LD that's quietly excluded from rich result eligibility.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)