A cache in front of a slow query works fine until a hot key expires. Then every request that arrives in the next few hundred milliseconds misses at the same moment and they all hit the database together. That's a cache stampede. The usual fixes are locks, background refresh, or probabilistic early expiration. The last one is the simplest and needs no coordination.
This post covers CachePulse, a single-file Python in-process cache (standard library only) that uses probabilistic early expiration (the "XFetch" approach), plus LRU capacity limits and built-in hit-rate and latency telemetry. Everything below was run locally with the shipped file.
The idea behind XFetch
Each entry remembers how long it took to compute (delta). On every read, the cache rolls a random number and recomputes early if:
-beta * delta * ln(random()) > remaining_ttl
Far from expiry the condition is almost never true. Close to expiry it becomes more and more likely, and more so for values that are expensive to compute. With many readers, usually one of them refreshes the key shortly before it expires, and the rest keep getting the cached value. beta tunes how aggressive this is: above 1 refreshes earlier, below 1 later.
In CachePulse that check is a few lines:
def _should_recompute_early(self, entry: CacheEntry) -> bool:
if entry.compute_duration <= 0.0:
return False
r = random.random() or 0.00001
early_threshold = -self.beta * entry.compute_duration * math.log(r)
return early_threshold > entry.remaining_ttl
Basic usage
The download contains main.py. I renamed it to cachepulse.py so I could import it:
import time
from cachepulse import CachePulseEngine
cache = CachePulseEngine(capacity=1000, default_ttl=30.0, beta=1.0)
def load_user(user_id):
time.sleep(0.05) # pretend this is a slow DB query
return {"id": user_id, "plan": "pro"}
for _ in range(200):
cache.get_or_compute("user:42", lambda: load_user(42))
s = cache.pulse()
print(f"hits={s.hit_count} misses={s.miss_count} hit_ratio={s.hit_ratio_pct:.1f}%")
print(f"p50={s.p50_latency_ms:.3f}ms p99={s.p99_latency_ms:.3f}ms saved={s.estimated_compute_saved_sec:.2f}s")
cache.shutdown()
hits=199 misses=1 hit_ratio=99.5%
p50=0.001ms p99=0.018ms saved=9.97s
saved adds up the recorded compute time of every hit, so 199 hits × ~50 ms is about 10 s of query time not spent. get_or_compute also takes a per-call ttl= and force_refresh=True.
What pulse() gives you
pulse() returns a TelemetrySnapshot dataclass. It has the key count and capacity, hits, misses, hit ratio, evictions, early refreshes ("stampede preventions"), p50/p95/p99 latency over a rolling window of the last 2,000 operations, and estimated compute time saved. It's a plain object, so you can log it or expose it from a /metrics endpoint:
snap = cache.pulse()
logger.info("cache", extra=dataclasses.asdict(snap))
The built-in load test
Running the file directly starts 8 threads hammering 10 hot keys for 8 seconds. Each miss costs a 30 ms simulated query, the TTL is 1.8 s, and capacity is 50. A live status line updates in place. The final summary on my machine:
RUNTIME EXECUTION COMPLETED (DIAGNOSTIC SUMMARY)
Total Operations Run : 11,878
Cache Hit Rate : 99.11%
Anti-Stampede Early Computes: 79 events prevented
Latency Profile (Sliding) : p50: 0.011ms | p95: 0.015ms | p99: 30.090ms
Backend Compute Saved : 354.47 seconds of thread time
Read it like this. The 106 misses are where the 30 ms p99 comes from, and 79 of them were early refreshes rather than hard expiries. Those 79 happened while the old value was still being served to everyone else. Your numbers will vary with CPU and timing, which is the point of running it on your own hardware.
Eviction and TTL
Two mechanisms keep memory bounded:
-
LRU on insert. When the cache is full, the least recently used key is dropped. I checked this with
capacity=3and four keys: the first key was evicted andeviction_countwent up. -
A background reaper thread. It wakes up once a second and removes expired entries. Call
cache.shutdown()to stop it cleanly.
Handling writes
There's no delete() method. When you change the underlying data, refresh the key right away:
db.update_plan(42, "team")
cache.get_or_compute("user:42", lambda: load_user(42), force_refresh=True)
If a stale value for up to ttl seconds is acceptable, a short TTL is enough on its own.
Limitations
- In-process only. Each Python process has its own cache. With gunicorn workers or several pods, each keeps its own copy and its own stats. It doesn't replace Redis or Memcached as a shared cache.
- Early refresh helps hot keys, not cold ones. If a key is missing or already expired and many threads ask for it at the same instant, each of them computes it. In a test with 8 threads hitting an empty key at once, the loader ran 8 times. If that matters, warm critical keys at startup or wrap the loader in your own per-key lock.
- No key invalidation API beyond
force_refresh(see above). - The telemetry window is the last 2,000 operations, so the percentiles describe recent traffic, not all-time.
FAQ
Is it thread-safe? Yes. Store access and the telemetry counters are protected by locks, and the compute function runs outside the lock so slow loaders don't block hits. I ran 8 writer threads while calling pulse() in a loop for several seconds without problems.
Does it work with asyncio? get_or_compute is synchronous. Use it for sync code or wrap it with asyncio.to_thread.
Dependencies? None. Python 3 and the standard library.
CachePulse is a one-time purchase that includes the source and README: get CachePulse ($29). More details are on the CachePulse product page.
Top comments (6)
Targeted invalidation can reduce broad purges, but the important failure case is a dropped or reordered mutation event. I would test the dependency graph with a sequence containing an update, an invalidation retry, and a late old event, then assert that a versioned cache entry cannot be overwritten by stale work. That makes the claimed stale-read boundary measurable rather than depending on the control plane alone.
Spot on! This is precisely the kind of edge case that breaks distributed control planes in production. Out-of-order event delivery (like a delayed older mutation arriving after a fresh invalidation retry) is the ultimate test for cache consistency.
Implementing monotonic versioning or sequence IDs on cache entries (effectively rejecting writes where incoming_version <= current_version) is indeed the cleanest way to neutralize late-arriving stale work. I love the idea of turning this exact sequence into a formal integration test for the dependency graph. Thanks for sharing this brilliant architectural hardening tip!
A strong integration fixture could make the ordering explicit: apply version 12, delay version 11, retry the invalidation for 12, then deliver 11 and assert that both the cache value and dependency metadata still point to 12. Capturing the accepted and rejected sequence IDs in the trace would make a later regression diagnosable.
Thanks , that fixture makes the failure mode concrete. Applying 12, delaying 11, retrying the invalidation for 12, then delivering 11 and asserting both the value and the dependency metadata still point to 12 is exactly the sequence I'd want before trusting any versioned invalidation path. Logging accepted vs rejected sequence IDs in the trace is a solid idea for diagnosing later regressions too. Today CachePulse stays simpler: in-process XFetch early refresh plus force_refresh, with no dependency graph or versioned writes, so that control-plane edge case isn't in scope. If I ever add keyed invalidation beyond force_refresh, I'd start with that exact ordering test so the stale-read boundary is measurable rather than assumed.
Thanks for clarifying the current scope. For the in-process path, a smaller useful fixture would be concurrent callers around early expiry: count refresh executions and check what each caller receives while a refresh is in flight. That tests the stampede boundary without implying the library already has distributed ordering semantics.
That's a great fit for the actual scope, thanks. Concurrent callers around early expiry is exactly where a stampede guard either holds or quietly doesn't. I'll add a fixture along those lines: N threads hitting the same key inside the early-refresh window, asserting the refresh runs once and recording what each caller gets back while it's in flight (stale value vs. fresh). Turning "prevents stampedes" into a counted number is the right way to keep the claim honest.