DEV Community: Bigboss6797976

The 6-Hour Ghost: How I Exorcised a Cache Avalanche with Async Reconnection and a Bloom Filter

Bigboss6797976 — Wed, 15 Jul 2026 03:14:28 +0000

*At 3 AM, the P99 latency on the monitoring screen spiked from 50ms to 3.8 seconds, then automatically dropped back down after two minutes. No errors, no garbage collection, and the Redis panel appeared normal. It was like a ghost, punctually strangling our order system every six hours.

This was my most unforgettable debugging experience—not because it was particularly complex, but because all conventional methods failed, and the truth ultimately lay in the "tacit understanding" between TCP Keepalive and the cloud SLB.
Problem Description (with images preferred)
• Service: Core e-commerce order system, peak QPS 20,000
• Symptoms: Severe fluctuations every 360 minutes (precise as a clock)
• P99 latency: 50ms → 3800ms (76 times)
• Timeout rate: 0.1% → 5%, approximately 2000 orders lost each time
• Duration: Automatically recovers after approximately 2 minutes

The team initially suspected a "scheduled task" or "JVM Full GC," but these were ruled out after investigation.

Detective Work (How I found the "ghost")

Stack Extraction Before the next fluctuation, I continuously dumped 50 thread snapshots using jstack. The comparison revealed: 60% of worker threads were blocked at RedisConnectionFactory.getConnection(), and all were waiting for the same lock.
Elimination Method
Redis Monitoring: CPU, memory, and connection counts are all normal.
GC Logs: No Full GC.
Network: No packet loss.
The Truth Revealed Redis was not configured with an idle timeout, but the cloud provider's SLB forcibly disconnected idle connections every 360 minutes. When the connection was actively closed by the server, thousands of concurrent requests simultaneously discovered the connection failure and flooded into getConnection() to compete for the lock and synchronously reconnect.

This is the "thundering herd effect"—all threads queue to rebuild connections, the database is compromised, and recovery only occurs when a new connection pool is established.

Remedial Solutions (Three Prescriptions)
Prescription 1: Asynchronous Reconnection + Local Degradation (Stopping the Bleeding)
· Only the first request triggers asynchronous reconstruction (CompletableFuture.supplyAsync)
· Other requests immediately return to the Caffeine local cache (allowing 5 seconds of old data)
· Retry uses exponential backoff + random jitter to avoid reconnection storms


// Core logic pseudocode

if (connection.isBroken()) {

if (atomicFlag.compareAndSet(false, true)) {

CompletableFuture.runAsync(this::reconnect);
}
return fallbackCache.get(key); // Non-blocking!
}

Result: 99.99% reconnection success rate, eliminating regular jitter for the first time.

Prescription Two: Bloom Filter + Proactive Refresh (Root Cause Solution)

· Bloom filter (96MB memory supports 100 million keys) blocks 90% of non-existent key penetration attacks.

· Set cache TTL to 10 minutes, with a background scheduled task asynchronously refreshing 2 minutes in advance to ensure keys never expire simultaneously.

Result: Redis hit rate increased from 85% to 99.2%, database QPS decreased by 94%.

Prescription Three: Timeout Circuit Breaker + Fallback (Ultimate Insurance)
· Set Redis operation timeout to 15ms + Hystrix circuit breaker.
· Automatically switch to local cache after timeout and record degradation metrics.
Result: Even if Redis completely crashes, the order creation process remains available.

Results Comparison (Data Speaks)
Indicators Before Optimization After Optimization Improvement
P99 Latency (Steady State) 50ms 32ms ↓36%
**Jitter Peak 3800ms No Jitter Eliminated
Cache Hit Rate 85% 99.2% ↑14.2%
CPU Average Load 75% 45% ↓40%
Timeout Rate 0.1% 0% Completely Zeroed

The monitoring curve changed from jagged peaks to a straight green line.

Lessons Learned (For Followers)

No anomalies in monitoring ≠ System health—You must delve into the thread stack and infrastructure layer.
Connection pools are not a panacea—the "silent disconnection" of external dependencies is the most insidious killer.
Cache invalidation should be "proactive" rather than "passive"—use preloading instead of expired eviction.
Always have fallback—even old data is better than timeout. After this fix, I deleted the "emergency restart script" prepared by the operations team—because it was no longer needed. That 6-hour ghost has finally been completely sealed away. --- AcknowledgementsThis is a submission for Weekend Challenge: Passion Edition* ## What I Built  ## Demo  ## Code  ## How I Built It  ## Prize Categories

That's what the Finish-Up-A-Thon is all about.

Bigboss6797976 — Sun, 07 Jun 2026 23:49:59 +0000

This is a submission for the GitHub Finish-Up-A-Thon Challenge
What I Built PayFusion AI is a full-stack intelligent payment aggregation system that supports multi-channel payment with one QR code using platforms like Alipay, WeChat, USDT, and PayPal. It integrates automatic transfers, fund distribution, QR code parsing, and AI-driven real-time payment monitoring. This project started out as a simple Flask QR code generator but gradually evolved into a complete payment automation platform. Key features include: Multi-channel payment with one QR code: Dynamically generates aggregated QR codes and intelligently routes users to the corresponding payment channels based on their scanning environment. AI fund monitoring Agent: Monitors payment status in real-time, automatically confirms fund arrival, and triggers subsequent workflows. Dual-platform architecture: Flask Web management backend + Telegram Bot mobile control. Automatic transfer engine: Supports automatic collection and fund distribution of USDT on the TRON network. Offline signature and blind signature: Complete blockchain security testing module. The entire project was developed and deployed entirely in a Termux Android terminal, eventually pushed to GitHub and automated using Railway/Render.
Demo GitHub repository: Bigboss6797976/ailipay
Telegram Bot: @PayFusionBot (demo environment) Web panel: https://ailipay-production.up.railway.app (demo environment) Screenshot preview: Web-side aggregated QR code generation interface Telegram Bot real-time payment notifications Fund diversion rule configuration panel Development environment in Termux
Turning point: Systematically "cleaning up the mess" I decided not to rewrite but to fix it—this is what Finish-Up-A-Thon is all about. First step: Environmental cleanup

Thoroughly clean up the corrupted lock files on Hardhat

rm -rf node_modules package-lock.json npm cache clean --force npm install
Step 2: Termux Compatibility Modifications Install Termux-specific compilation dependencies for Pillow: pkg install libjpeg-turbo libpng Replace psutil with native Python os and subprocess to rewrite the system monitoring module Downgrade TronWeb to v4 stable and refactor all address generation logic Install the zbar system library: pkg install zbar and set LD_LIBRARY_PATH

Step 3: Deploy Pipeline Fixes Fix Flask's RGBA color parsing errors by switching to hexadecimal color values Generate a one-click deployment script, railway_fix.sh, to automate CI/CD from Termux → GitHub → Railway Create a streamlined requirements file for Render, removing dependencies with compilation conflicts

Step 4: Architecture Upgrade Split the messy scripts into modular structures: bot.py (TG side), app.py (Web side), payment/ (core), crypto/ (on-chain) Introduce the aiogram 3.7+ asynchronous framework and rewrite all Telegram interaction logic Add environment variable management to completely decouple sensitive information from the code

Current Status Today, this project has become a production-ready system: ✅ Passed all unit tests with zero errors ✅ Stably running on Railway and Render ✅ Telegram Bot supports over 20 commands and 80+ feature modules ✅ Complete payment flow: scan code → payment → confirmation → automatic transfer → notification ✅ Codebase has evolved from "unrunnable" to "maintainable and scalable"

The Biggest Takeaway What this project taught me wasn’t just a specific technology stack, but how to deliver engineering solutions in resource-limited environments. When the development environment is just an Android phone, when the compiler throws its 50th error, when the documentation says "not supported on this platform"—choosing to keep debugging instead of giving up—that’s what "Finishing-Up" really means.

To you guys who are also developing on phones/tablets, if you're struggling with Termux, remember these tips: 1. Use $HOME as your build directory and copy it to shared storage to avoid permission issues. 2. If you run into compilation errors, check the system libraries first - pkg install usually works before pip install. 3. Lock down dependency versions since Termux isn't standard Linux, and the latest versions often mean incompatibility. 4. Write a one-click script to combine "push → deploy → check" into one command to reduce mental stress. Thanks to GitHub and DEV for organizing this challenge. Completing one project is more valuable than starting ten. If you're also cleaning up messes, keep going - the release date is just around the corner. If you're interested in this project, feel free to Star it or submit an Issue on GitHub!