There's a moment in almost every growing web application's life when someone adds caching to a slow endpoint, the response time drops dramatically, and everyone celebrates. Then, two weeks later, a user emails support because they updated their profile and their old name is still showing everywhere. Or an inventory count reads 50 units in stock while the database says 0. Or a published article doesn't appear for some users for minutes after going live.
Caching and consistency pull in opposite directions by nature. The faster you want to serve data, the further it travels from the source of truth and the more opportunities there are for it to drift. Large-scale systems don't solve this tension by picking one side. They manage it deliberately, with explicit trade-offs at each layer. This article breaks down how they do it.
Understanding the Consistency Problem First
Before reaching for solutions, it helps to be precise about what "consistency" actually means in a caching context, because the word gets used loosely.
Strong consistency means every read reflects the most recent write, everywhere, every time. Achievable, but expensive. It usually means skipping the cache or invalidating it synchronously on every write, which defeats the performance purpose.
Eventual consistency means that given enough time without new writes, all reads will converge on the same value. Most large-scale caching systems accept eventual consistency as the baseline and work to minimize the window during which stale data is served.
Read-your-own-writes consistency is a weaker, often more practical guarantee: after a user modifies data, their subsequent requests should reflect that change, even if other users still see stale state. This is the minimum standard most user-facing applications actually need.
Knowing which of these your application requires, per feature, is the design decision that drives everything downstream.
The Core Strategies Large Systems Use
1. Cache-Aside (Lazy Loading)
The most common pattern, and the right starting point for most applications. The application checks the cache first; on a miss, it reads from the database, populates the cache, and returns the result. Writes go directly to the database, and the cache entry is either invalidated or left to expire.
def get_user_profile(user_id: str) -> dict:
cache_key = f"user:profile:{user_id}"
# Check cache first
cached = redis_client.get(cache_key)
if cached:
return json.loads(cached)
# Cache miss — read from DB
profile = db.query("SELECT * FROM users WHERE id = %s", user_id)
# Populate cache with a TTL
redis_client.setex(cache_key, 300, json.dumps(profile)) # 5-minute TTL
return profile
def update_user_profile(user_id: str, data: dict) -> None:
db.execute("UPDATE users SET ... WHERE id = %s", user_id)
# Invalidate — don't update in place
redis_client.delete(f"user:profile:{user_id}")
The critical choice here is invalidate on write, not update. Updating a cache entry in place during a write introduces a window where a concurrent read between the DB write and cache update can repopulate stale data, a race condition that's hard to reproduce but real under load.
The weakness of cache-aside is that invalidation only works if every write path knows which cache keys to invalidate. In a system with multiple services writing to the same data, this coupling becomes brittle fast.
2. Write-Through Caching
Every write goes to the cache and the database simultaneously, keeping them in sync by construction. Reads always hit the cache, which is always current.
async function updateProductStock(productId, quantity) {
const cacheKey = `product:stock:${productId}`;
// Write to DB first
await db.query(
'UPDATE products SET stock = $1 WHERE id = $2',
[quantity, productId]
);
// Immediately update cache — same transaction window
await redisClient.set(cacheKey, quantity, { EX: 3600 });
}
This eliminates stale reads after writes but introduces a different problem: write amplification. Every write now requires two operations, both of which must succeed. If the cache write fails after a successful DB write, you're inconsistent. If you write to cache first and the DB write fails, you're serving data the database doesn't have.
The pattern handles this most reliably when cache and DB writes are treated as a unit — either with a transaction wrapper (where the DB supports distributed transactions) or by accepting that rare partial failures are caught by TTL expiration rather than prevented.
Write-through works well for data with high read-to-write ratios where stale reads are genuinely costly: product inventory, pricing, user permissions.
3. Event-Driven Cache Invalidation
As systems grow and multiple services write to the same underlying data, coordinating invalidation through the application layer breaks down. A cleaner approach is Change Data Capture (CDC): let the database emit events on every data change, and have a cache invalidation service listen to those events and purge the relevant keys.
[PostgreSQL] → [Debezium CDC connector] → [Kafka topic: db.changes] → [Cache Invalidation Service] → [Redis UNLINK]
The cache invalidation service consumes change events and computes which cache keys map to the changed row. Because it listens to the database's own write-ahead log, it catches changes from every path application code, admin tools, data migrations, background jobs without any of them needing to know about the cache.
# cache_invalidation_consumer.py
def handle_db_change(event: CDCEvent):
table = event.table
row_id = event.after.get("id")
if table == "users":
redis_client.unlink(f"user:profile:{row_id}")
redis_client.unlink(f"user:summary:{row_id}")
elif table == "products":
redis_client.unlink(f"product:detail:{row_id}")
redis_client.unlink(f"product:stock:{row_id}")
# Also invalidate any list caches that include this product
redis_client.unlink("products:featured")
This is how large platforms like LinkedIn and Shopify handle cache invalidation at scale, the cache layer has no knowledge of write sources, and write sources have no knowledge of the cache.
The trade-off is operational complexity: Kafka, Debezium, and the invalidation consumer all need to be maintained, and event processing latency (usually milliseconds, occasionally more) means there's still a brief eventual consistency window.
4. Cache Versioning and the "Generation" Pattern
For read-heavy data that changes infrequently but must be consistent when it does change, cache versioning avoids key-based invalidation entirely.
Instead of invalidating a key, increment a version number. The cache key includes the version, so old entries simply become unreachable, they expire naturally without any explicit delete needed.
def get_site_config():
# The "version" is stored as a separate key and incremented on every config change
version = redis_client.get("site:config:version") or "1"
cache_key = f"site:config:v{version}"
cached = redis_client.get(cache_key)
if cached:
return json.loads(cached)
config = db.query("SELECT * FROM site_config")
redis_client.setex(cache_key, 86400, json.dumps(config))
return config
def update_site_config(data):
db.execute("UPDATE site_config SET ...")
# Bump the version — old cache entries become orphaned
redis_client.incr("site:config:version")
This is particularly useful for user-facing settings, feature flag configurations, and global site data where reads vastly outnumber writes. No distributed delete is required on write — old entries simply age out.
The Thundering Herd Problem
When a popular cache key expires and hundreds of requests hit the origin simultaneously to repopulate it, you've created a self-inflicted load spike. This is the thundering herd problem, and it's what has brought down production databases when a cache warms up after a deployment.
Two standard mitigations:
Mutex/distributed lock on cache miss:
import redis
import threading
def get_with_mutex(cache_key, fetch_fn, ttl=300):
cached = redis_client.get(cache_key)
if cached:
return json.loads(cached)
lock_key = f"lock:{cache_key}"
lock = redis_client.set(lock_key, "1", nx=True, ex=10) # NX = only set if not exists
if lock:
# This process won the lock — fetch and populate
try:
data = fetch_fn()
redis_client.setex(cache_key, ttl, json.dumps(data))
return data
finally:
redis_client.delete(lock_key)
else:
# Lost the race — wait briefly and retry from cache
time.sleep(0.05)
return get_with_mutex(cache_key, fetch_fn, ttl)
Stale-while-revalidate: Serve the stale entry for a short additional window while a background process refreshes it, eliminating the gap entirely. Redis doesn't support this natively, but it can be approximated by storing both the data and a separate "refresh needed" marker.
Read-Your-Own-Writes: The Practical Minimum
For user-facing applications where eventual consistency is otherwise acceptable, read-your-own-writes is often the only consistency guarantee that actually matters. A user who changes their email address and immediately sees the old one on the next page will file a bug report. A user who doesn't know another user's profile updated a second ago won't notice.
The implementation is typically session-affinity at the cache layer: route reads for a given user to the same cache node or replica for a short window after a write. Alternatively, bypass the cache entirely for a user's own data for a configurable period after they make a change.
def get_user_data(user_id, requesting_user_id):
# If the user is reading their own data and recently modified it, skip cache
recently_modified_key = f"modified:{requesting_user_id}"
if user_id == requesting_user_id and redis_client.exists(recently_modified_key):
return db.query("SELECT * FROM users WHERE id = %s", user_id)
return get_from_cache_or_db(f"user:{user_id}")
def update_user_data(user_id, data):
db.execute("UPDATE users SET ... WHERE id = %s", user_id)
redis_client.delete(f"user:{user_id}")
# Mark this user as having a recent write for 30 seconds
redis_client.setex(f"modified:{user_id}", 30, "1")
When Caching Architecture Goes Wrong: A Real-World Pattern
A common failure mode in growing systems: a team implements cache-aside correctly for their main data, then adds a background job that writes to the database directly — bypassing the application layer and therefore bypassing cache invalidation. The cache serves stale data indefinitely until the TTL expires, and since TTL was set to 24 hours for performance reasons, the window is a full day.
This exact scenario is why event-driven invalidation via CDC eventually wins at scale over application-layer invalidation. It doesn't matter how the data changed — admin panel, background job, manual SQL, migration script — the database event fires and the cache is invalidated.
Teams building distributed systems encounter this progression regularly. Those doing website development in Glenview, Illinois for high-traffic applications often see this pattern play out in the first few months after launch — the application-layer invalidation holds initially, then breaks the first time a data migration or admin script writes directly to the database.
Choosing the Right Strategy per Layer
There's no single right answer — the practical approach is mapping each data type to the appropriate strategy:
| Data Type | Recommended Strategy | Reason |
|---|---|---|
| User sessions | Cache-aside, short TTL (15–30 min) | Frequently invalidated, security-sensitive |
| Product catalog | Write-through or event-driven | High read volume, accuracy matters |
| User-generated content | Cache-aside + read-your-writes | Consistency matters to the author, eventual for others |
| Config / feature flags | Versioned cache | Low write frequency, global reads |
| Aggregate counts (likes, views) | Approximate caching acceptable | Exact count rarely critical |
| Inventory / stock | Write-through, short TTL | Stale data has real business cost |
The teams who manage caching well at scale aren't using one strategy — they're using all of them, applied deliberately to the data types where each makes sense. Engineers doing website development in Plainfield, Illinois for e-commerce clients typically handle this by auditing each data entity separately and assigning a caching tier with explicit consistency requirements documented alongside the implementation.
Observability: You Can't Fix What You Can't See
Any production caching layer needs metrics or it's flying blind:
- Cache hit ratio — below 80% on high-traffic endpoints usually means the cache isn't helping as much as expected
- Key eviction rate — high eviction means the cache is undersized or TTLs are too short
- Stale read rate — requires application-level instrumentation where feasible, checking whether a cached value differs from the DB on a sample of reads
- Invalidation lag — for event-driven systems, the time between a DB write and cache invalidation; spikes here indicate consumer lag
Redis exposes most of these through INFO stats and keyspace notifications. Wire them into your monitoring stack (Datadog, Prometheus, or equivalent) and alert on hit ratio drops, which are usually the first visible signal of a caching bug before users report data inconsistency.
Summary
Large-scale websites don't choose between caching and consistency, they choose their consistency model per data type and implement caching strategies that honor those choices. Cache-aside with TTL is the right starting point for most features. Write-through handles high-stakes read-heavy data. Event-driven CDC invalidation via tools like Debezium scales cleanly as write sources multiply. Cache versioning simplifies global configuration. And read-your-own-writes is often the only consistency guarantee that materially affects user experience.
The mistake that causes most production caching bugs isn't a wrong algorithm choice it's treating caching as a single, uniform layer rather than a deliberate set of trade-offs applied per domain.
Top comments (0)