The 6-Hour Ghost: How I Exorcised a Cache Avalanche with Async Reconnection and a Bloom Filter

#devchallenge #weekendchallenge #bugsmash #performance

*At 3 AM, the P99 latency on the monitoring screen spiked from 50ms to 3.8 seconds, then automatically dropped back down after two minutes. No errors, no garbage collection, and the Redis panel appeared normal. It was like a ghost, punctually strangling our order system every six hours.

This was my most unforgettable debugging experience—not because it was particularly complex, but because all conventional methods failed, and the truth ultimately lay in the "tacit understanding" between TCP Keepalive and the cloud SLB.
Problem Description (with images preferred)
• Service: Core e-commerce order system, peak QPS 20,000
• Symptoms: Severe fluctuations every 360 minutes (precise as a clock)
• P99 latency: 50ms → 3800ms (76 times)
• Timeout rate: 0.1% → 5%, approximately 2000 orders lost each time
• Duration: Automatically recovers after approximately 2 minutes

The team initially suspected a "scheduled task" or "JVM Full GC," but these were ruled out after investigation.

Detective Work (How I found the "ghost")

Stack Extraction Before the next fluctuation, I continuously dumped 50 thread snapshots using jstack. The comparison revealed: 60% of worker threads were blocked at RedisConnectionFactory.getConnection(), and all were waiting for the same lock.
Elimination Method
Redis Monitoring: CPU, memory, and connection counts are all normal.
GC Logs: No Full GC.
Network: No packet loss.
The Truth Revealed Redis was not configured with an idle timeout, but the cloud provider's SLB forcibly disconnected idle connections every 360 minutes. When the connection was actively closed by the server, thousands of concurrent requests simultaneously discovered the connection failure and flooded into getConnection() to compete for the lock and synchronously reconnect.

This is the "thundering herd effect"—all threads queue to rebuild connections, the database is compromised, and recovery only occurs when a new connection pool is established.

Remedial Solutions (Three Prescriptions)
Prescription 1: Asynchronous Reconnection + Local Degradation (Stopping the Bleeding)
· Only the first request triggers asynchronous reconstruction (CompletableFuture.supplyAsync)
· Other requests immediately return to the Caffeine local cache (allowing 5 seconds of old data)
· Retry uses exponential backoff + random jitter to avoid reconnection storms


// Core logic pseudocode

if (connection.isBroken()) {

if (atomicFlag.compareAndSet(false, true)) {

CompletableFuture.runAsync(this::reconnect);
}
return fallbackCache.get(key); // Non-blocking!
}

Result: 99.99% reconnection success rate, eliminating regular jitter for the first time.

Prescription Two: Bloom Filter + Proactive Refresh (Root Cause Solution)

· Bloom filter (96MB memory supports 100 million keys) blocks 90% of non-existent key penetration attacks.

· Set cache TTL to 10 minutes, with a background scheduled task asynchronously refreshing 2 minutes in advance to ensure keys never expire simultaneously.

Result: Redis hit rate increased from 85% to 99.2%, database QPS decreased by 94%.

Prescription Three: Timeout Circuit Breaker + Fallback (Ultimate Insurance)
· Set Redis operation timeout to 15ms + Hystrix circuit breaker.
· Automatically switch to local cache after timeout and record degradation metrics.
Result: Even if Redis completely crashes, the order creation process remains available.

Results Comparison (Data Speaks)
Indicators Before Optimization After Optimization Improvement
P99 Latency (Steady State) 50ms 32ms ↓36%
**Jitter Peak 3800ms No Jitter Eliminated
Cache Hit Rate 85% 99.2% ↑14.2%
CPU Average Load 75% 45% ↓40%
Timeout Rate 0.1% 0% Completely Zeroed

The monitoring curve changed from jagged peaks to a straight green line.

Lessons Learned (For Followers)

No anomalies in monitoring ≠ System health—You must delve into the thread stack and infrastructure layer.
Connection pools are not a panacea—the "silent disconnection" of external dependencies is the most insidious killer.
Cache invalidation should be "proactive" rather than "passive"—use preloading instead of expired eviction.
Always have fallback—even old data is better than timeout. After this fix, I deleted the "emergency restart script" prepared by the operations team—because it was no longer needed. That 6-hour ghost has finally been completely sealed away. --- AcknowledgementsThis is a submission for Weekend Challenge: Passion Edition* ## What I Built  ## Demo  ## Code  ## How I Built It  ## Prize Categories

DEV Community

The 6-Hour Ghost: How I Exorcised a Cache Avalanche with Async Reconnection and a Bloom Filter

The team initially suspected a "scheduled task" or "JVM Full GC," but these were ruled out after investigation.

This is the "thundering herd effect"—all threads queue to rebuild connections, the database is compromised, and recovery only occurs when a new connection pool is established.

Result: Redis hit rate increased from 85% to 99.2%, database QPS decreased by 94%.

The monitoring curve changed from jagged peaks to a straight green line.

Top comments (0)