DEV Community

Poetical Engine
Poetical Engine

Posted on

How I keep 13,000 kids' events fresh on Cloudflare Workers (and the 128 MB wall I hit)

Grassy Knees is a directory of real things to do with kids across 42 US metro areas. Its rule is simple: every listing comes from the organizer's own page and shows the date it was last checked. This post covers the parts that turned out to be interesting to build.

The daily refresh plan

Re-researching 13k listings every day would be wasteful, so a planner runs over the database first. It picks a few "deep" metros to research from scratch and a list of recheck items: events close to their date, sources that changed, listings that failed a check. Research agents (LLMs with a browser and fetch) read the organizers' pages and return JSON records. A full pass across all metros took 32 agents and about 6.3M tokens, and added 2,415 public listings.

The import gate

Agents are confident even when they're wrong, so nothing goes live without passing a gate. A record needs a street address, an official URL on the organizer's domain, and a date in range. It must not copy from another directory. Records that fail are logged as DROPs with the reason, which helps tune the next brief.

Library feeds are a firehose

Library event systems (BiblioCommons, LibCal, Communico) expose clean feeds, and libraries run more free family events than anyone. Left alone, they took over whole metros (Seattle was 100% library one day). The plan now carries a per-metro mix with a cap on the library share.

The 128 MB wall

The site is Next.js-style React rendered on Cloudflare Workers, with D1 for data, KV for page caches and R2 for images. Loading the full catalog to render a list page OOM'd the 128 MB isolate. The fix was precomputed per-metro slices plus streaming. A side quest: an orphaned per-version page cache had grown to 356k keys (~15 GB) in KV before I noticed.

Markdown twins for AI assistants

Every page has a .md twin (append .md to the URL) and the site has an llms.txt, so assistants can quote real listings with their source links instead of inventing events.

It's free to browse. If you're a parent, I'd love to know what's missing in your city. If you're a builder, I'm happy to go deeper on any of this in the comments.

Top comments (1)

Collapse
 
onecoderegister profile image
Andrew Cluev •

The “last checked” date is such a good detail! 👍