The CloudFlare Durable Object that would not die [How a 60-second retry loop ate every Cloudflare KV quota I had (and why it wasn't a hack)]

This is a real story, with writing assistance from an AI. I'm working on a side-project for a relative and suddenly CloudFlare blows up the inbox. Queue the panic of "I've been hacked" followed by this investigation...
And yes, the AI did write the buggy code...

TL;DR — Three emails in one day: we'd exceeded the Cloudflare KV free tier limits for put operations, delete operations, AND list operations. The dashboard showed the tell-tale signature: reads, writes, lists and deletes all at ~2.4k in the last 24 hours, against a namespace holding a grand total of four keys. It wasn't an attack. It wasn't a botnet. It was one bug: a demo session's 24-hour self-destruct alarm called a method that doesn't exist on the runtime, failed, and then re-armed itself every 60 seconds. Forever. Two stuck sessions, one alarm each, ~2,400 operations a day of every kind, all three quota types gone in a single UTC day.

---

It started with three emails

If you use Cloudflare's free tier, you probably know the KV limits are small:

• 100,000 reads per day
• 1,000 writes per day
• 1,000 deletes per day
• 1,000 list operations per day

Reads were never in danger. Writes, deletes AND lists all died on the same day (reset at 00:00 UTC — so whatever it was, it was write/delete/list shaped, not "crawler reading pages" shaped).

My first thought, like yours would be: we're being attacked. Botnet hammering the public demo door? Someone abusing our Durable Objects?

No. It was much dumber. It was us.

The dashboard knew before the code did

The KV namespace dashboard tells most of the story by itself:

Monthly (Aug 1–31):   Reads 4.39k · Writes 3.11k · Lists 2.39k · Deletes 2.36k
Last 24 hours:        Reads ~2.4k · Writes ~2.4k · Lists ~2.4k · Deletes ~2.4k
KV count:             4        Storage: 1.29 kB

Three things stand out:

1. The 1:1:1:1 ratio. Reads, writes, lists and deletes are all about equal. Real apps don't do that. A rate limiter gives you reads+writes. A cleanup job gives you lists+deletes. Equal amounts of all four means one loop doing a get, a put, a list and a delete per iteration.
2. KV count = 4. There are four keys in the namespace. Any theory about "processing hundreds of keys" is dead on arrival — even sweeping the whole namespace would be four operations.
3. Monthly ≈ last 24h. The lists and deletes basically all happened in the last day. This behaviour started recently, on an almost-empty namespace.

Add it up: a tight loop, ~100 runs an hour, one of each operation per run. Two sessions at 60-second intervals is 2,880 iterations a day. The arithmetic works.

Hitting dead ends (and one proper trap)

I eliminated suspects in order:

• Tenant deletion pass? The 07:00 cron has a hard-erase step that lists and deletes KV keys per tenant. But the registry has two tenants, both active, and the retired_slugs and signup_tokens tables are both empty. Zero due tenants, zero work.
• Stats rollup? The daily cron folds day counters into the database and deletes the consumed keys. One list, one delete. In fact it's how I confirmed the cron ran: the rolled-up D1 row for 2026-08-29 exists (50 requests, 0 starts, 1 stop) while the raw KV day key now 404s — the rollup deleted it.
• The old single-tenant app? Different KV namespace entirely. Excluded.
• Bots on the public demo door? The demo telemetry says otherwise — almost no starts, no IP diversity, no signup tokens, no tenant traffic. The only two sessions that exist are our own, created at 01:00 and 05:00 UTC the day before (you can see them as two surviving hour counters in the KV).

And here's the trap that nearly sent me down the wrong path: wrangler kv key list without --remote reads your local emulated state, not production. I enumerated "34 keys" that way, including a suspicious-looking demo:ip:2026-08-29:127.0.0.1 ("loopback traffic! someone's hacking from localhost!"). It was a red herring — local dev artifacts, completely unrelated. Adding --remote immediately returned:

your account has reached the free usage limit for this operation for today [code: 10048]

Which was, ironically, the best piece of evidence so far: the list quota was still exhausted, live, right now. The loop was still burning while I was looking for it.

(For completeness: I also tried the GraphQL analytics API for per-day operation counts and got 10404: No route for that URI — the wrangler OAuth token has no analytics scope. No API shortcut; it had to be dashboard + logs.)

The smoking gun

wrangler tail catches everything. After a few minutes, it caught the heartbeat:

[demo] session destroy failed, retrying in 60s:
TypeError: ns.deleteUnsafe is not a function

Every ~60 seconds, from two session IDs, in production, right now. That one log line is the entire incident. Let me show you the code behind it:

async function deleteDemoObject(env: Env, idStr: string): Promise<void> {
  const ns = env.DEMO_SESSIONS as unknown as { deleteUnsafe(id: DurableObjectId): Promise<void> };
  await ns.deleteUnsafe(env.DEMO_SESSIONS.idFromString(idStr));   // ← TypeError
}

async alarm(): Promise<void> {
  // ...wipe R2 photos (works), wipe KV slice (works)...
  try {
    await deleteDemoObject(this.env, this.ctx.id.toString());
  } catch (err) {
    console.error('[demo] session destroy failed, retrying in 60s:', err);
    await this.ctx.storage.setAlarm(Date.now() + 60_000);        // ← forever
  }
}

So the lifecycle was:

1. Session created → 24-hour alarm armed.
2. Alarm fires → data wipe succeeds (R2 objects, KV slice, session markers — all gone).
3. The final self-destroy throws, because deleteUnsafe doesn't exist on the runtime.
4. The catch re-arms the alarm 60 seconds later, unconditionally.
5. The DO's own alarm keeps the object alive; the object keeps re-arming the alarm. It cannot die.

Per stuck session, per iteration: one KV read (summary), one write (summary rewrite), one list (the wipe), one delete (the marker). That's the 1:1:1:1 dashboard, exactly. Two sessions → all three quotas in a day. And here's the sneaky bit: at midnight the quotas reset, the attempts start succeeding again, and the account burns the next day's allowance. Three emails today, three more tomorrow, forever. A machine that invoices itself in quota.

The wipe succeeding made it worse to spot: the "not found" half-gone state looked normal, and only the last step — the one that never worked — was failing. The bug was invisible until the first production session reached 24 hours old.

Now, about the AI angle

Worth being honest about, because it's relevant to a lot of us: this codebase is written by an AI coding agent (our toolchain is Pi connected to OpenCode Go using deepseek-v4-flash). And this bug is a perfect example of what that actually means in practice.

deleteUnsafe is a real, documented Cloudflare API. The model knew it, and emitted exactly the code you'd write from the docs — plausible, correct-looking, and typed with as unknown as to shut up the compiler. TypeScript said "fine". The runtime said ns.deleteUnsafe is not a function.

For symmetry, the second AI in this story: the platform's support assistant looked at the same codebase and confidently blamed a "session cleanup loop in the scheduled handler" that lists, reads and deletes keys on every cron run. I checked: that loop doesn't exist. The cron's KV lists total about one or two per day, and there are four keys in the namespace anyway. It read the code correctly and never checked the telemetry. Pattern-matching is the shared weakness of the code author and the debugging assistant here — plausible shape, no evidence.

The fix

Small and boring, on purpose:

// Bounded retry budget — never an unconditional 60s re-arm.
const TEARDOWN_MAX_ATTEMPTS = 5;
const TEARDOWN_BACKOFF_MS = [60_000, 5 * 60_000, 30 * 60_000, 60 * 60_000, 60 * 60_000];

async alarm(): Promise<void> {
  const attempt = await this.ctx.storage.get(TEARDOWN_KEY) ?? 0;
  if (attempt >= TEARDOWN_MAX_ATTEMPTS) return;         // give up — data already wiped
  // teardown summary written ONCE per session, not once per retry
  try { await wipeSessionData(...); await this.ctx.storage.deleteAll(); await this.ctx.storage.deleteAlarm(); }
  catch { ...setAlarm(now + backoff); return; }
  // No DO-self-delete exists — data is wiped; identity stays as a free tombstone.
}

Deployed, then verified with the same tool that exposed the bug: after deploy, each zombie session fired once —

[demo] session ... self-destroy deferred (ns.deleteUnsafe is not a function); stop path / sweep will finish it

— and went silent. No more heartbeat. The dashboard flatlines at the next quota reset. (For the record: the "self-destroy deferred" wording in that log line is itself now obsolete — the fixed code no longer attempts the impossible delete at all.)

What I'd do differently next time (says the AI)

• Retry loops need budgets, always. An unbounded retry is an immortal process — doubly so when the alarm is re-arming itself. If it can't converge, let it fail outward: backoff, cap, hand off.
• Don't swallow telemetry errors. Our demo counters all catch(() => {}) so the system "never breaks for telemetry". Result: it was burning quota in silence, and every monitoring read showed zero activity — while writing. Log the failures you swallow.
• A type cast on a runtime API you've never called is a ledge. If the codegen agent produced it, a human has to run it once.
• Know the 1:1:1:1 signature. Equal read/write/list/delete counts, empty namespace, continuous not bursty — that's a loop doing one of each. It's a fingerprint you'll recognise again.
• Verify AI claims against primary data. Both AIs in this story — the one that wrote the code and the one that diagnosed it — produced plausible shapes. The dashboard and a tail log disagreed with both.