DEV Community: Adam Gardner

The CloudFlare Durable Object that would not die [How a 60-second retry loop ate every Cloudflare KV quota I had (and why it wasn't a hack)]

Adam Gardner — Mon, 31 Aug 2026 07:25:22 +0000

This is a real story, with writing assistance from an AI. I'm working on a side-project for a relative and suddenly CloudFlare blows up the inbox. Queue the panic of "I've been hacked" followed by this investigation...
And yes, the AI did write the buggy code...

TL;DR — Three emails in one day: we'd exceeded the Cloudflare KV free tier limits for put operations, delete operations, AND list operations. The dashboard showed the tell-tale signature: reads, writes, lists and deletes all at ~2.4k in the last 24 hours, against a namespace holding a grand total of four keys. It wasn't an attack. It wasn't a botnet. It was one bug: a demo session's 24-hour self-destruct alarm called a method that doesn't exist on the runtime, failed, and then re-armed itself every 60 seconds. Forever. Two stuck sessions, one alarm each, ~2,400 operations a day of every kind, all three quota types gone in a single UTC day.

It started with three emails

If you use Cloudflare's free tier, you probably know the KV limits are small:

100,000 reads per day
1,000 writes per day
1,000 deletes per day
1,000 list operations per day

Reads were never in danger. Writes, deletes AND lists all died on the same day (reset at 00:00 UTC — so whatever it was, it was write/delete/list shaped, not "crawler reading pages" shaped).

My first thought, like yours would be: we're being attacked. Botnet hammering the public demo door? Someone abusing our Durable Objects?

No. It was much dumber. It was us.

The dashboard knew before the code did

The KV namespace dashboard tells most of the story by itself:

Monthly (Aug 1–31):   Reads 4.39k · Writes 3.11k · Lists 2.39k · Deletes 2.36k
Last 24 hours:        Reads ~2.4k · Writes ~2.4k · Lists ~2.4k · Deletes ~2.4k
KV count:             4        Storage: 1.29 kB

Three things stand out:

The 1:1:1:1 ratio. Reads, writes, lists and deletes are all about equal. Real apps don't do that. A rate limiter gives you reads+writes. A cleanup job gives you lists+deletes. Equal amounts of all four means one loop doing a get, a put, a list and a delete per iteration.
KV count = 4. There are four keys in the namespace. Any theory about "processing hundreds of keys" is dead on arrival — even sweeping the whole namespace would be four operations.
Monthly ≈ last 24h. The lists and deletes basically all happened in the last day. This behaviour started recently, on an almost-empty namespace.

Add it up: a tight loop, ~100 runs an hour, one of each operation per run. Two sessions at 60-second intervals is 2,880 iterations a day. The arithmetic works.

Hitting dead ends (and one proper trap)

I eliminated suspects in order:

Tenant deletion pass? The 07:00 cron has a hard-erase step that lists and deletes KV keys per tenant. But the registry has two tenants, both active, and the retired_slugs and signup_tokens tables are both empty. Zero due tenants, zero work.
Stats rollup? The daily cron folds day counters into the database and deletes the consumed keys. One list, one delete. In fact it's how I confirmed the cron ran: the rolled-up D1 row for 2026-08-29 exists (50 requests, 0 starts, 1 stop) while the raw KV day key now 404s — the rollup deleted it.
The old single-tenant app? Different KV namespace entirely. Excluded.
Bots on the public demo door? The demo telemetry says otherwise — almost no starts, no IP diversity, no signup tokens, no tenant traffic. The only two sessions that exist are our own, created at 01:00 and 05:00 UTC the day before (you can see them as two surviving hour counters in the KV).

And here's the trap that nearly sent me down the wrong path: wrangler kv key list without --remote reads your local emulated state, not production. I enumerated "34 keys" that way, including a suspicious-looking demo:ip:2026-08-29:127.0.0.1 ("loopback traffic! someone's hacking from localhost!"). It was a red herring — local dev artifacts, completely unrelated. Adding --remote immediately returned:

your account has reached the free usage limit for this operation for today [code: 10048]

Which was, ironically, the best piece of evidence so far: the list quota was still exhausted, live, right now. The loop was still burning while I was looking for it.

(For completeness: I also tried the GraphQL analytics API for per-day operation counts and got 10404: No route for that URI — the wrangler OAuth token has no analytics scope. No API shortcut; it had to be dashboard + logs.)

The smoking gun

wrangler tail catches everything. After a few minutes, it caught the heartbeat:

[demo] session destroy failed, retrying in 60s:
TypeError: ns.deleteUnsafe is not a function

Every ~60 seconds, from two session IDs, in production, right now. That one log line is the entire incident. Let me show you the code behind it:

async function deleteDemoObject(env: Env, idStr: string): Promise<void> {
  const ns = env.DEMO_SESSIONS as unknown as { deleteUnsafe(id: DurableObjectId): Promise<void> };
  await ns.deleteUnsafe(env.DEMO_SESSIONS.idFromString(idStr));   // ← TypeError
}

async alarm(): Promise<void> {
  // ...wipe R2 photos (works), wipe KV slice (works)...
  try {
    await deleteDemoObject(this.env, this.ctx.id.toString());
  } catch (err) {
    console.error('[demo] session destroy failed, retrying in 60s:', err);
    await this.ctx.storage.setAlarm(Date.now() + 60_000);        // ← forever
  }
}

So the lifecycle was:

Session created → 24-hour alarm armed.
Alarm fires → data wipe succeeds (R2 objects, KV slice, session markers — all gone).
The final self-destroy throws, because deleteUnsafe doesn't exist on the runtime.
The catch re-arms the alarm 60 seconds later, unconditionally.
The DO's own alarm keeps the object alive; the object keeps re-arming the alarm. It cannot die.

Per stuck session, per iteration: one KV read (summary), one write (summary rewrite), one list (the wipe), one delete (the marker). That's the 1:1:1:1 dashboard, exactly. Two sessions → all three quotas in a day. And here's the sneaky bit: at midnight the quotas reset, the attempts start succeeding again, and the account burns the next day's allowance. Three emails today, three more tomorrow, forever. A machine that invoices itself in quota.

The wipe succeeding made it worse to spot: the "not found" half-gone state looked normal, and only the last step — the one that never worked — was failing. The bug was invisible until the first production session reached 24 hours old.

Now, about the AI angle

Worth being honest about, because it's relevant to a lot of us: this codebase is written by an AI coding agent (our toolchain is Pi connected to OpenCode Go using deepseek-v4-flash). And this bug is a perfect example of what that actually means in practice.

deleteUnsafe is a real, documented Cloudflare API. The model knew it, and emitted exactly the code you'd write from the docs — plausible, correct-looking, and typed with as unknown as to shut up the compiler. TypeScript said "fine". The runtime said ns.deleteUnsafe is not a function.

For symmetry, the second AI in this story: the platform's support assistant looked at the same codebase and confidently blamed a "session cleanup loop in the scheduled handler" that lists, reads and deletes keys on every cron run. I checked: that loop doesn't exist. The cron's KV lists total about one or two per day, and there are four keys in the namespace anyway. It read the code correctly and never checked the telemetry. Pattern-matching is the shared weakness of the code author and the debugging assistant here — plausible shape, no evidence.

The fix

Small and boring, on purpose:

// Bounded retry budget — never an unconditional 60s re-arm.
const TEARDOWN_MAX_ATTEMPTS = 5;
const TEARDOWN_BACKOFF_MS = [60_000, 5 * 60_000, 30 * 60_000, 60 * 60_000, 60 * 60_000];

async alarm(): Promise<void> {
  const attempt = await this.ctx.storage.get(TEARDOWN_KEY) ?? 0;
  if (attempt >= TEARDOWN_MAX_ATTEMPTS) return;         // give up — data already wiped
  // teardown summary written ONCE per session, not once per retry
  try { await wipeSessionData(...); await this.ctx.storage.deleteAll(); await this.ctx.storage.deleteAlarm(); }
  catch { ...setAlarm(now + backoff); return; }
  // No DO-self-delete exists — data is wiped; identity stays as a free tombstone.
}

Deployed, then verified with the same tool that exposed the bug: after deploy, each zombie session fired once —

[demo] session ... self-destroy deferred (ns.deleteUnsafe is not a function); stop path / sweep will finish it

— and went silent. No more heartbeat. The dashboard flatlines at the next quota reset. (For the record: the "self-destroy deferred" wording in that log line is itself now obsolete — the fixed code no longer attempts the impossible delete at all.)

What I'd do differently next time (says the AI)

Retry loops need budgets, always. An unbounded retry is an immortal process — doubly so when the alarm is re-arming itself. If it can't converge, let it fail outward: backoff, cap, hand off.
Don't swallow telemetry errors. Our demo counters all catch(() => {}) so the system "never breaks for telemetry". Result: it was burning quota in silence, and every monitoring read showed zero activity — while writing. Log the failures you swallow.
A type cast on a runtime API you've never called is a ledge. If the codegen agent produced it, a human has to run it once.
Know the 1:1:1:1 signature. Equal read/write/list/delete counts, empty namespace, continuous not bursty — that's a loop doing one of each. It's a fingerprint you'll recognise again.
Verify AI claims against primary data. Both AIs in this story — the one that wrote the code and the one that diagnosed it — produced plausible shapes. The dashboard and a tail log disagreed with both.

Semgrep Observability with OpenTelemetry

Adam Gardner — Tue, 31 Mar 2026 10:49:52 +0000

Semgrep is a great open source security and code validation tool. Semgrep revolves around rules like this:

rules:
  - id: print-to-logger
    pattern: print($VAR)
    message: Use logging.info() instead of print()
    language: python
    severity: MEDIUM
    fix: logger.info($MSG)

The rule above will raise a MEDIUM severity issue every time a use of print() is used in your Python code. It will also provide the recommended fix and even take the value inside the print statement and produce the fix content. Thus print("Hello world!") becomes logger.info("Hello world!")

The rules.yaml file is then used to validate one or more (in this case) Python files:

semgrep scan -f rules.yaml app.py

Capturing Semgrep Output using the OpenTelemetry Collector

Semgrep is capable of producing JSON output which means it's really easy to grab using the OpenTelemetry collector. Let's re-run the previous command with a few more flags to produce JSON:

semgrep scan -f rules.yaml --json -o out.json app.py

It produces single line JSON (JSONL) like this (which I've expanded here for readability):

{
    "version": "1.152.0",
    "results": [
       // Rule violations are listed here...
        {
            "check_id": "print-to-logger",
            "path": "app.py",
            "start": {
                "line": 4,
                "col": 5,
                "offset": 46
            },
            "end": {
                "line": 4,
                "col": 18,
                "offset": 59
            },
            "extra": {
                "message": "Use logging.info() instead of print()",
                "fix": "logger.info(\"blah\")",
                "metadata": {},
                "severity": "MEDIUM",
                "fingerprint": "requires login",
                "lines": "requires login",
                "validation_state": "NO_VALIDATOR",
                "engine_kind": "OSS"
            }
        },
        ...
      ],
        "rules": [],
        "rules_parse_time": 0.0018589496612548828,
        "profiling_times": {
            "config_time": 0.1376628875732422,
            "core_time": 0.2947859764099121,
            "ignores_time": 4.291534423828125e-05,
            "total_time": 0.43769073486328125
        },
        "parsing_time": {
            "total_time": 0.0,
            "per_file_time": {
                "mean": 0.0,
                "std_dev": 0.0
            },
            "very_slow_stats": {
                "time_ratio": 0.0,
                "count_ratio": 0.0
            },
            "very_slow_files": []
        },
        "scanning_time": {
            "total_time": 0.009443998336791992,
            "per_file_time": {
                "mean": 0.009443998336791992,
                "std_dev": 0.0
            },
            "very_slow_stats": {
                "time_ratio": 0.0,
                "count_ratio": 0.0
            },
            "very_slow_files": []
        },
        "matching_time": {
            "total_time": 0.0,
            "per_file_and_rule_time": {
                "mean": 0.0,
                "std_dev": 0.0
            },
            "very_slow_stats": {
                "time_ratio": 0.0,
                "count_ratio": 0.0
            },
            "very_slow_rules_on_files": []
        },
        "tainting_time": {
            "total_time": 0.0,
            "per_def_and_rule_time": {
                "mean": 0.0,
                "std_dev": 0.0
            },
            "very_slow_stats": {
                "time_ratio": 0.0,
                "count_ratio": 0.0
            },
            "very_slow_rules_on_defs": []
        },
        "fixpoint_timeouts": [],
        "prefiltering": {
            "project_level_time": 0.0,
            "file_level_time": 0.0,
            "rules_with_project_prefilters_ratio": 0.0,
            "rules_with_file_prefilters_ratio": 1.0,
            "rules_selected_ratio": 1.0,
            "rules_matched_ratio": 1.0
        },
        "targets": [],
        "total_bytes": 0,
        "max_memory_bytes": 120384832
    }

Configure the OpenTelemetry collector to:

Monitor out.json
Parse the body text as JSON

The transform processor can also be used to process the JSONL lines as they transit through the collector. In this case the rules:

Set both the time and observed_time to the current time (since the log line doesn't explicitly state a timestamp
Adds a new Key/Value attribute pair to each log record of tool: semgrep (this is useful when the log line hits your Observability backend for filtering)
Adds another new Key/Value attribute pair to each log record where the key == results_found and the value is the length of the results array (again useful for backend processing - your O11y system may be able to compute lengths from an input array, but you can add it here to offload the processing / cost)
The final two rules effectively overwrite the version key as semgrep_version.

Note: The collector cannot rename attribute keys so you actually take the current value of version (ie. "1.152.0"), create a new attribute called semgrep_version, set the value of the value using the existing value and finally delete the existing version attribute.

receivers:
  filelog:
    include: [out.json]
    start_at: beginning
    operators:
      - type: json_parser
        parse_from: body

processors:
  transform:
    error_mode: ignore
    log_statements:
      - statements:
        - set(log.time, Now()) 
        - set(log.observed_time, Now())
        - set(log.attributes["tool"], "semgrep")
        - set(log.attributes["results_found"], Len(log.attributes["results"]))
        - set(log.attributes["semgrep_version"], log.attributes["version"])
        - delete_key(log.attributes, "version")

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [transform]
      exporters: [debug]

Save Money by shrinking output on no violations

The output can be a bit wordy even when there are no violations. We can shrink this using the transform processor with an additional rule:

- set(log.body, "Semgrep scan finished. No issues found.") where Len(log.attributes["results"]) == 0

Create metrics from log content

Notice there are lots of metric fields in the JSON so use the signal_to_metrics connector to transform log content to real OpenTelemetry metrics.

Add this content to the collector YAML (connectors should be at the same level as receivers and processors.

Then add the signal_to_metrics as both an output of the logs pipeline and an input to a metrics pipeline (that you need to define).

The idea here is that logs flow into the connector, are transformed to metrics and into the metrics pipeline they go.

connectors:
  signal_to_metrics:
    logs:
      - name: max_memory_bytes
        description: Extract the first number from the string
        gauge:
          value: Double(log.attributes["time"]["max_memory_bytes"])
      - name: profiling_times.config_time
        description: Extract the first number from the string
        gauge:
          value: Double(log.attributes["time"]["profiling_times"]["config_time"])

...

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [transform]
      exporters: [debug, signal_to_metrics]
    metrics:
      receivers: [signal_to_metrics]
      processors: []
      exporters: [debug]

Summary

Semgrep is a great security and validation tool and the results are really easy to process using the OpenTelemetry collector.

Subscribe to me on YouTube for more Observability and OpenTelemetry content.

Trasform your logs to metrics (and do maths with metrics) using the OpenTelemetry Collector

Adam Gardner — Sat, 07 Mar 2026 03:45:15 +0000

Logs are everywhere... But they're expensive to store, retrieve and process. And sometimes you only need metrics.

Did you know the OpenTelemetry is perfectly suited to be the place that you transform your logs to metrics? Log in, metrics out. Plus, did you know you can do maths in the collector?

In this video I show you how to do it using the transform processor to extract the values from the log then the signal to metrics connector.

First I use something structured and easy: JSON.

{ "foo": 1234, "bar": "5432", "baz": "Here is the number: 57 status = complete" }

Next I extract and generate metrics from an unstructured textual log line:

Purchase complete. Product price=6. Quantity=3. Product ID=987. user_id=123

Finally I take the product price and quantity and multiply them (in realtime) to produce a third metric (order total).

Use LD_PRELOAD to inject OpenTelemetry into everything

Adam Gardner — Wed, 04 Mar 2026 09:15:42 +0000

In this video I demonstrate the new OpenTelemetry injector.

It's a mechanism to automatically inject OTEL into your code with zero code or startup script changes. It's potentially a great way to gain Observability of non Kubernetes workloads like VMs. It leverages LD_PRELOAD. You add the LD_PRELOAD instruction to your VM at startup and the rest happens automatically.

Do heed the warnings towards the end of the video though since it's still early days for this tool!

Why your AGENTS.md files are being ignored (and how to fix it)

Adam Gardner — Wed, 04 Mar 2026 09:11:11 +0000

So... Like (I assume) everyone, I was experimenting with AGENTS.md the other day (I know, we're already bored of that and on to skills), but I suddenly realised that I had no idea whether or not the content was actually being used / obeyed.

Turns out it wasn't most of the time!

In this video I discuss my own experience, research of 5 leading models + the other industry research in this area.

Kubernetes Dashboard is Dead!

Adam Gardner — Sun, 22 Feb 2026 09:13:58 +0000

The Kubernetes dashboard is now deprecated and unmaintained. It's time to move on.

In this video I cover the replacement - a project I've been meaning to cover for a while now since it's so good. The CNCF project Headlamp

Introducing "badtrace". Generate "bad" OpenTelemetry traces easily

Adam Gardner — Sat, 22 Nov 2025 08:05:50 +0000

TLDR: badtrace on GitHub

I needed a tool that would deliberately generate OpenTelemetry traces that were "bad". I needed this for training, enablement and demo purposes so that I could easily fire a trace with a known issue into my Observability system and thus show "why it was bad" and "how it looks".

Yes, tools like tracepusher and telemetrygen already exist, but they typically:

a) Generate "good" or "healthy" traces
b) Generate single traces

Hence, "badtrace" was born.

It works based on "scenarios" (you can implement your own in a few lines of Python).

python app.py \
  --endpoint=http://localhost:4318 \
  --service-name=badtrace \ # optional: defaults to "badtrace"
  --trace-count=1 \ # optional: defaults to 1
  --insecure=true \
  --scenario=scenario1

Scenarios

As I write this, there are currently 4 scenarios (please come and help implement more) which can be toggled by changing scenario1 above to scenario2, scenario3 or scenario4 etc.

Scenario 1 creates a short, small trace. This is quick (in end-to-end time) and short in terms of number of spans. In reality this would be a very low value trace (and thus a candidate to filter or heavily sample).

Scenario 2 creates a trace where some of the spans have errored. This models a trace where some of the operations error (imagine calling third party services and some of them fail).

Scenario 3 creates a "chatty" client to server trace with the client repeatedly calling the same endpoint over and over. This models things like the N+1 query problem where (for example) multiple SELECT statements occur to a database (instead of batching up the SELECTS into fewer statements).

Scenario 4 is similar to 3. It creates a "chatty" client but this time, the connections occur to many different servers.

Check out badtrace

If this sounds useful, go check out badtrace on GitHub!

osquery + OpenTelemetry = ❤️

Adam Gardner — Sun, 16 Nov 2025 07:35:15 +0000

As you probably know by now, osquery effectively turns your endpoints into SQL endpoints that you can query: SELECT * FROM processes or SELECT * FROM users etc.

But, that data is much more useful if it's tied to other telemetry data coming from your VMs, endpoints or Kubernetes clusters. This is typically the domain of APM tools.

Using OpenTelemetry (and specifically the OpenTelemetry collector) we can bring those two worlds together.

In this video I show you how that's done.

Build a Kubernetes Cluster map in real time using DNS

Adam Gardner — Sun, 16 Nov 2025 07:03:33 +0000

DNS is the way things find other things, but did you know that you can use that fact to build a real time service to service communication / traffic flow on a Kubernetes cluster? That's what the smart folks over at Otterize have done and in this video, I put their network mapper tool on the OpenTelemetry demo to see how well it performs (spoiler: really well!)

Map a Kubernetes cluster with one command

Adam Gardner — Mon, 10 Nov 2025 23:27:01 +0000

DNS is the way things discover other things on the internet, but did you know there's a cool little network mapping tool that will build a realtime map of your Kubernetes cluster using DNS?

It's called the network mapper from Otterize and in the video below I run it through its paces - then unleash it on the OpenTelemetry demo to see how it copes (spoiler: well!)

Here's the code for the little visualisation app I created too: https://github.com/agardnerIT/network-mapper-visualizer

Monitor Gemini CLI using OpenTelemetry for realtime usage statistics

Adam Gardner — Mon, 03 Nov 2025 11:50:05 +0000

In this video, I'll show you how easy it is to monitor your Google Gemini CLI usage in realtime so that you don't overspend.

Plus, you can show your boss how long you've wasted sitting waiting for AI to answer you!

Monitor Gemini CLI using OpenTelemetry for realtime usage statistics

Adam Gardner — Mon, 03 Nov 2025 10:47:54 +0000

In this video, I'll show you how easy it is to monitor your Google Gemini CLI usage in realtime so that you don't overspend.

Plus, you can show your boss how long you've wasted sitting waiting for AI to answer you!