quick build writeup, because i haven't seen many people run a directory this way and it's been surprisingly nice to live with.
the site is vibeking, a directory of vibe-coded products (stuff people built mostly by prompting AI). around 2,500 entries right now, each with its own page, and the whole thing runs as ONE cloudflare worker with a d1 database behind it. no framework, no build step, no origin server. here's how it works and the two things that bit me.
one entry, not one page. the trick that makes a big directory maintainable is you never build pages, you build data. every product is a row in d1 (name, tagline, url, category, tags). one render function turns any row into a full page: title, meta, og tags, json-ld, internal links to related products, all of it. so the 2,000th product costs the same effort as the 5th. the worker matches /product/:slug, pulls the row, renders. that's basically the whole architecture.
d1 at the edge is great until you think about it wrong. it's sqlite, and for a read-heavy directory that's perfect. the mistake i made early was doing too much per request. put indexes on whatever you sort/filter by (category, votes, created_at) on day one, not after it's slow.
now the two things that actually bit me, because this is the useful part:
1. the soft-404 that quietly poisons your seo. my worker had a catch-all that, for any path it didn't recognize, just served the homepage with a 200. felt harmless. it is not. google crawls a bunch of guessed/junk urls, every one returns 200-homepage, and google decides "this site returns the same page for everything" and starts distrusting your urls. for a directory whose entire seo bet is thousands of unique pages, that's poison. the fix is boring but critical: any unmatched path must return a real 404. probe your own site with a fake url and confirm you get a 404. i'd fixed this on a couple of my other sites and STILL had to catch it here.
2. the cold-start problem is real and your page count doesn't save you. here's the humbling bit. i thought 2,500 pages was a moat. it isn't, on its own. a brand-new domain with 2,500 thin pages is 2,500 pages google mostly ignores, because you haven't earned trust yet. page count is potential energy, not traffic. what turns it on is the slow stuff: real internal linking, genuinely useful individual pages, being something an AI assistant would recommend, and earning a few real links from outside. i'm in the middle of that part now and it's the opposite of fun, which is probably why it's the part most of us skip.
if you're building anything directory-shaped on workers, those are the two i'd save you: return real 404s, and don't mistake a big sitemap for traction.
the site if you want to see the pattern in the wild: vibeking.fun. happy to answer anything about the worker/d1 setup in the comments.
thanks for reading.
Top comments (0)