A Slow Warm Call Proved the Cache Contract Failed, Not the Cause

Originally published on hexisteme notes.

I run a small serverless backend on Cloudflare Workers. One of its endpoints assembles a schedule board by walking 29 pages of an upstream API, then caches the assembled result at the edge. The cache was in the design record, in the code, and in the cache-control header of every response it returned. It had been live in production.

Then a verification step asked it to prove itself. The requirement had been written down in plain words — the second call must come back from the stored value — so it got a test: call the endpoint, call it again, time the second call. The expected number was about 0.16 seconds.

The second call took 2.9 seconds. It had re-walked all 29 upstream pages. I ran it again: 2.9 seconds again. That was enough to reject the behavioral claim — “the second call comes from stored work” — but not enough to name the cause.

What the timing proved — and what it did not

My first postmortem blamed the *.workers.dev hostname. I wrote that Cloudflare's Cache API had no effect there and treated a future custom domain as the switch that would make the same calls work. That explanation was clean, platform-shaped, and false.

Cloudflare's official documentation history contains the falsifier. On 2025-03-05 — more than a year before this incident — Cloudflare merged EW-6697 Remove workers.dev Cache API restriction. The commit message says the runtime bug had already been fixed about a year earlier and the docs were the stale part. The current Cache API documentation excludes dashboard-editor and Playground previews, and Workers fronted by Cloudflare Access; it does not exclude workers.dev deployments.

So what survives from the incident? Two back-to-back calls repeated the expensive upstream walk, and the later per-isolate memo stopped that repetition. What does not survive is the leap from “warm behavior missing” to “hostname caused the miss.” The probe did not preserve the direct result of cache.match(), the exact read and write key, the final cache-relevant headers, or the Cloudflare data-center suffix for both requests. Without those, cache-key mismatch, response eligibility, locality, a skipped read path, and an actual Cache API miss remain live explanations.

The broader claim about five cache call sites also has to shrink. They shared an API pattern, but I did not instrument every path and therefore did not prove that none had ever served a hit. The honest record is narrower: the schedule-board contract failed its warm-call test on the measured path, and my first root-cause story failed an official-history check.

Nothing broke, because the layer underneath was paying

Every request that backend had ever served in production ran with zero caching, and that should have produced a signal somewhere. It produced none, and the reason generalizes further than the bug does.

Underneath the cache sat a second protection: per-day spend counters in the SQL store, acting as circuit breakers over the paid upstream. Underneath that, the free-tier quotas of the upstreams themselves. Those two absorbed the entire request volume without complaint. The system looked healthy because it was healthy — just not for the reason the design record gave.

That is the trap in stacked protections. Layer A (the cache) is supposed to remove load. Layer B (the breaker, the quota, the retry budget, the autoscaler) is supposed to bound the damage when load arrives anyway. When A is ineffective on a path, B covers for it silently, and the only trace is that B's budget is what is actually burning. Absence of failure is evidence about B. It is not evidence about A.

The question that would have caught this in review: if this layer were deleted entirely, what would I observe? If the honest answer is "nothing, for a while," you do not have a monitoring gap — you have an unverifiable claim, and it stays unverifiable until you measure that layer directly. Caches are the common case, but the same silence covers a retry policy that never retries, a rate limiter that is never reached, and a fallback path that is never taken. Any layer whose success signature is identical to a normal response can be missing for months.

A cache you have not timed is an assumption

Here is everything I had accepted as evidence before that timing test:

• The put() call is in the code, on the right path, with the right key.
• The response carries cache-control with a sensible TTL.
• A test asserts the header is present.

None of those observes a hit. The last one is the most seductive, because it is green and it says "cache" in the test name. Reading the cache-control header off a response your own code just built tells you what your code wrote into a header. It says nothing about whether any cache stored the response, and nothing about whether a later request read it back.

Three things actually observe a hit:

1. Time a warm call. Two identical requests, back to back. Write down the expected number before running it — the point of the expectation is that a miss becomes a failed assertion instead of a shrug at a slow response.
2. Emit an explicit hit marker. Have the read path set x-cache: HIT or MISS at the point where the lookup returns, not where the response is assembled. Then a hit is directly observable in a curl, in a log line, and in a smoke test.
3. Count hits and misses. A hit rate is a number you can chart and alert on. A cache with no hit-rate metric is one whose failure mode is silence.

Timing and a marker should be paired, because either one alone is ambiguous. This is the minimal shape I should have deployed before naming a cause:

function marked(response: Response, source: "cache-api" | "origin"): Response {
  const headers = new Headers(response.headers);
  headers.set("x-cache-source", source);
  return new Response(response.body, {
    status: response.status,
    statusText: response.statusText,
    headers,
  });
}

const cached = await caches.default.match(cacheKey);
if (cached) return marked(cached, "cache-api");

const response = await buildResponse();
await caches.default.put(cacheKey, response.clone());
return marked(response, "origin");

Then run the same URL twice from the same client and record four fields for each call: total time, x-cache-source, the CF-Ray data-center suffix, and a hash of the normalized cache key written by the Worker log. A slow second call with x-cache-source: origin is a witnessed miss. A fast call without that marker is only a latency observation. A cache-api hit that is still slow falsifies “the expensive work sits behind this lookup” and sends the investigation to network time or a different code path.

The reason this got caught at all was an accident of wording worth stealing on purpose. The requirement had been stated as behavior ("serve it from the stored value"), not as an optimization. Optimizations do not get assertions; behaviors do. If a cache matters — for cost, for a paid upstream, for a latency budget — write it down as a behavior with a number attached, and it gets a test that is capable of failing.

Why a cache layer quietly runs at a zero percent hit rate

The exact cause in this incident remains unverified. The general failure — a cache that is present, correct-looking, and not observed hitting — has a short list of usual suspects. When a warm call comes back cold, walk this before touching the cache code:

Cause	The check that settles it
Environment where caching is a no-op	Check the current platform contract for the exact surface. Cloudflare currently names dashboard-editor and Playground previews, plus Access-fronted Workers — not workers.dev generally
Cache key mismatch	Log the key on write and on read, then diff them; query-string order, trailing slashes, and method all move the key
Vary	Any Vary on a header that differs per client (User-Agent, Accept-Encoding, Cookie) fragments the entry; Vary: * disables storage outright
Set-Cookie on the response	Shared caches generally refuse to store a response that sets a cookie
Cache-Control: no-store, private, no-cache, or max-age=0	Print the final response headers; a framework or middleware default may be overwriting what you set
TTL shorter than the gap between requests	Compare the TTL against real inter-request time; a 60-second entry on a page requested every five minutes hits approximately never
Personalization or auth on the path	Authorization headers and per-user bodies are uncacheable by design; the fix is splitting the cacheable part out, not a longer TTL
Method or status not cacheable	POST responses and many non-200 statuses are not stored by default
Locality	A hit in one node, region, or isolate is a miss in the next; thin traffic spread across many nodes looks exactly like a broken cache
Write path runs, read path does not	A feature flag, an early return, or an error branch that skips the lookup gives you a cache that only ever fills

Every row can produce the identical symptom — a warm call as slow as a cold one — which is exactly why the measurement comes first and the theory second. My mistake was stopping after the measurement rejected the contract and letting a stale platform rule choose the cause. The next probe must make the read branch, key, headers, and locality observable.

What I replaced it with

The mitigation that restored the required behavior was a per-isolate in-memory memo. Measured on the deployed worker: 1.19 seconds cold, 0.157 seconds warm — close to the number the original test had been expecting all along. That result shows that repeated upstream work was eliminated within one isolate. It does not retroactively prove why the Cache API path failed to produce the same behavior.

That store is strictly weaker than an edge cache, and it is worth being precise about how. It lives inside one isolate. It dies when that isolate is recycled. A request routed to a different instance pays full price. It is a smaller promise than the one the design record made — but it is a promise I can measure, which the larger one turned out not to be.

The caches.default calls stayed in the code, but “dormant until a custom domain” is no longer an honest annotation. A custom domain was later added; that does not retroactively prove the old hostname diagnosis or prove that current calls hit. Until the marker-and-key probe runs, those calls are unverified, not dead and not trusted. That distinction matters to the next person who greps for them.

Limits

The replacement — an in-process memo — only makes sense where a cold miss is survivable and the data is shared across requests rather than per-user. It is not a global cache, and none of those properties transfer automatically.

The claim I would defend is narrower than "your cache is broken." It is that a cache hit is something you can only learn by observing the read branch, and that a warm-call timing failure rejects a behavior without identifying its cause. A stack of protections can conceal an ineffective layer for as long as the layer below it has budget.

The diagnosis now has explicit falsifiers. If instrumented workers.dev calls return x-cache-source: cache-api on the second request, the old hostname diagnosis is false — as the official history already predicts. If both workers.dev and the custom domain miss with the same normalized key and eligible headers in the same data center, the cause lies elsewhere in the read/write path. And a near-zero hit rate is not by itself a defect: low traffic, short TTLs, and many data centers can produce one legitimately. Timing finds the failed contract; markers and preserved inputs localize it.

More notes at hexisteme.github.io/notes.