An account dashboard may look inexpensive when it has one viewer. The problem shows up when everyone opens it during an incident and the cached value has just expired. Each request sees the expiry and starts its own external lookup. A slow dependency gets more work at exactly the wrong time.
This tutorial was generated with AI. The code and results were checked locally with synthetic inputs.
I’d use a similar rule for any low-frequency account lookup, including a read-only balance observation. Let one reader refresh the cached value and give the others an existing observation if it is still within an acceptable age. After that age, return an unavailable state. A balance that has never been read is unavailable from the start.
Separate freshness from the maximum stale age
In the example, an observation is fresh for five seconds and may still be shown for 30 seconds. These short thresholds keep the fixture easy to read. In a deployed system I’d choose intervals appropriate for the monitor. I’d also show the age and status in the dashboard so an operator can tell that the value they see is from an earlier observation.
I use two locks for distinct purposes. A small lock guards the cached snapshot. A separate refresh lock identifies the one reader allowed to invoke the loader. The loader runs outside the snapshot lock, so a follower can obtain the previous value without waiting for the refresh. The example uses a monotonic clock for elapsed age within the process. Python threading primitives.
from concurrent.futures import ThreadPoolExecutor
from threading import Event, Lock
class Cache:
def __init__(self, clock):
self.clock = clock
self.snapshot = None
self.data_lock = Lock()
self.refresh_lock = Lock()
def current(self):
with self.data_lock:
return self.snapshot
def view(self, snapshot):
if snapshot is None:
return ("unavailable", None)
age = self.clock() - snapshot[1]
if age > 30:
return ("unavailable", None)
state = "fresh" if age < 5 else "stale"
return (state, snapshot[0])
def get(self, load):
state, value = self.view(self.current())
if state == "fresh":
return (state, value)
if not self.refresh_lock.acquire(blocking=False):
return self.view(self.current())
try:
state, value = self.view(self.current())
if state == "fresh":
return (state, value)
try:
amount = load()
except TimeoutError:
return self.view(self.current())
with self.data_lock:
self.snapshot = (amount, self.clock())
return ("fresh", amount)
finally:
self.refresh_lock.release()
now = [0]
cache = Cache(lambda: now[0])
assert cache.get(lambda: 1200) == ("fresh", 1200)
now[0] = 6
started, release = Event(), Event()
def controlled_refresh():
started.set()
assert release.wait(5), "Test did not release refresh"
return 1100
def unexpected_refresh():
raise AssertionError("Second reader called the loader")
with ThreadPoolExecutor(max_workers=1) as pool:
owner = pool.submit(cache.get, controlled_refresh)
try:
assert started.wait(5), "Refresh did not start"
assert cache.get(unexpected_refresh) == ("stale", 1200)
finally:
release.set()
assert owner.result() == ("fresh", 1100)
def timeout():
raise TimeoutError
now[0] = 15
assert cache.get(timeout) == ("stale", 1100)
now[0] = 37
assert cache.get(timeout) == ("unavailable", None)
print("One refresh owner, stale fallback, and expiry verified.")
The two events in the test control when the loader starts and finishes. That lets the test place another reader inside the refresh window without a sleep. While the first reader is held, the second gets the stale value and does not call the loader. Once the held refresh finishes, a later timeout can use the new value until it passes the maximum age.
Keep the scope of the guarantee visible
The locks coordinate threads inside one Python process. They don’t coordinate multiple service replicas. I’d use a shared cache with an atomic lease, or a dedicated monitoring job that writes observations for the dashboard to read, if many processes need the same value. In the dedicated-job design, dashboard traffic can’t multiply external requests at all.
I’d also add a pause after a failed refresh. This example limits the number of simultaneous calls, but sequential visitors could still trigger repeated timeouts while a dependency is down. In production, a short recovery interval tied to the monitor’s cadence can protect the dependency and make alerts easier to interpret. Test that behavior with a controlled clock and a loader that fails repeatedly before putting a live account behind it.
Top comments (0)