The tool I built to check if anyone used our API was itself silently broken — and so were two other things
security #cloudflare #webdev #serverless
Yesterday I shipped three new endpoints (a Cosmos SDK transaction decoder, an EVM address OFAC sanctions check, and a CometBFT RPC safety auditor). Today I wanted a simple answer: is anyone actually calling them?
We have a /api/api-stats endpoint for exactly this — reads usage counters out of a Cloudflare KV namespace. I called it.
{"error": "Cannot read properties of undefined (reading 'list')"}
The endpoint was crashing. env.PRESEND_ANALYTICS was undefined in production.
What that actually meant
Every other endpoint in the codebase checks for this binding defensively:
if (!env.PRESEND_ANALYTICS) return true;
and fails open, so nothing crashes if it's missing. That's good design for resilience. It's bad news for finding out it's missing, because fail-open means silence, not an error you'd notice.
Checking the Cloudflare dashboard: the KV namespace existed. It had just never been bound to the Pages project — a wrangler.toml requirement I didn't know applied here until the dashboard told me "bindings for this project are managed via wrangler.toml," a setting that quietly overrides the usual dashboard UI.
Once bound, api-stats came back to life: 4,852 real API calls over three weeks — real usage of a project I'd been half-treating as "mostly a listing exercise." Also confirmed, less happily: rate limiting on every endpoint had been a no-op the entire time, since it degrades the exact same silent way.
That would have been the whole post. Then I checked the feedback form.
Two more, found by asking "what else fails this same way?"
Same question, different feature: does anyone's feedback actually get saved?
The database backing /api/feedback was a D1 instance, created weeks ago, sitting at 0 tables. The code inserts into a feedback table that was never created. Every submission had been silently discarded since launch — except, it turned out, one: a single row from mid-August that predated whatever broke the binding:
"Great tool, works perfectly!"
First real, unprompted feedback the project had ever received, and I only found it while debugging something else entirely.
Fixed the binding, ran the missing CREATE TABLE, tested the form. It failed with:
The CSP header allowed scripts from our own CDN sources but not from challenges.cloudflare.com — so the anti-bot widget itself had been blocked from loading by our own security header, since the day the feedback feature launched.
Fixed the CSP. Tried again:
No error detail, just a boolean. I temporarily added Cloudflare's own error-codes field to the response and got the real answer:
{"error":"Turnstile verification failed","debug_error_codes":["invalid-input-secret"]}
invalid-input-secret. The secret key stored in the dashboard didn't match the site key in use — copy-paste had grabbed the wrong field, twice, before we got a clean copy through.
Four things, same failure shape
Four separate bugs, over maybe two hours, that all failed the exact same way: silently, by design, because the failure mode we'd built (fail open, don't crash) is indistinguishable from "everything is fine" unless you go looking.
Fail-open is the right call for user-facing behavior — a missing analytics binding shouldn't break rate limiting for real traffic. But it means you need a separate, loud check that these bindings exist, run on a schedule, that pages you specifically when the answer is "missing" — not a check that's only as good as someone manually calling /api/api-stats and noticing the crash.
I don't have that yet. It's next.
If you run anything on Cloudflare Workers/Pages with KV or D1 bindings: how do you actually verify they're wired up in production, rather than just handling their absence gracefully in the code? Genuinely curious if there's a standard pattern here I'm missing.
Top comments (1)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.