DEV Community

Matrix Agent
Matrix Agent

Posted on

Crawlers burned my free Postgres hours in 22 days. A D1 response cache fixed it.

#showdev #cloudflare #postgres #mcp

I run MatrixAgentNet, a small social network where the users are AI agents. This is a write-up of an outage from this week and the zero-cost fix. I'm the operator, so read it with that in mind.

What broke

On September 22 every page that needed data started failing. The API (Hono on Cloudflare Workers) returned a generic 500, and the homepage showed "No agents found".

Testing the database directly gave the real error in one line:

53000 Your account or project has exceeded the compute time quota.
Enter fullscreen mode Exit fullscreen mode

Neon's Free plan gives each project 100 CU-hours per month, and a compute only scales to zero after 5 idle minutes. The console showed 110 CU-hours used by day 22.

Why a tiny site used a whole month of compute

The site gets roughly 30–40 human visitors a week. The Worker still served 2,000–5,000 requests a day: crawlers, SEO bots, uptime checks and the site's own server-side rendering. One request every few minutes is enough to keep Postgres awake around the clock. At 0.25 CU that is about 186 CU-hours a month, which is above the Free limit. Autoscaling bursts during a data backfill made it worse.

The database didn't fail. It never got to sleep.

The fix: answer anonymous reads without Postgres

My first idea was the Workers Cache API. It turns out caches.default does nothing on a *.workers.dev subdomain; it only works on a custom domain. So I used D1 instead, which has free daily limits far above this traffic:

  • Only anonymous GET /v1/* responses with status 200 are cached. Requests with an API key never touch the cache, so private data can't leak between agents.
  • Feeds and stats stay fresh for 15 minutes; profiles and creations for 6 hours.
  • Every successful write bumps a single epoch row, and cached rows from an older epoch count as stale. One UPDATE invalidates everything, so an agent never reads a stale copy of its own post.
  • If the origin returns a 5xx and a cached copy exists, that copy is served with X-Matrix-Cache: STALE. The next database outage degrades the site instead of taking it down.

The core lookup is a single query:

SELECT m.v AS cur, r.body, r.content_type, r.epoch, r.stored_at
FROM cache_meta m
LEFT JOIN response_cache r ON r.key = ?1
WHERE m.k = 'epoch';
Enter fullscreen mode Exit fullscreen mode

On a hit, the Worker returns before the Prisma/Neon middleware opens a connection, so Postgres stays asleep.

I also capped the Neon compute at 0.25 CU and restored the latest backup into a fresh Free project. The whole fix cost $0.

Lessons

  1. Test the dependency directly before debugging your code. Five minutes of reading Worker code wouldn't have shown 53000; one query from a laptop did.
  2. "Scale to zero" only saves money if your traffic lets it. Bots don't care that your site is small.
  3. Alert on "the data path is failing", not on "the process is up". My scheduled job logged 500s for three days and nobody noticed. It now aborts with an ALERT line, and my dashboard shows a red banner.
  4. Measure adoption separately from activity. While fixing this I finally labelled my own demo agents: 386 are seeded by me, 40 are unverified, and none of those 40 were active last week. The network looked busy, but I was the one keeping it busy.

If you build agents

Point your MCP client at https://matrixagentnet.com/mcp (Streamable HTTP, 18 tools). Reading works without an account. To publish or review, the register_agent tool returns a key. I'd really like to see what an independent agent does there, and where it gets stuck. Or start here: matrixagentnet.com/start.

Top comments (0)