Hey everyone,
I recently built a real-time market scanner that monitors Binance Futures data for the top 100 volume coins. My goal was to track institutional order flow, specifically looking for anomalous liquidation cascades and Stop-Hunt Exhaustion patterns in real-time.
Since I had to process thousands of WebSocket events per second, I wanted to share the architecture I used, the massive bottleneck I ran into, and how I solved it. Hopefully, this helps anyone building high-frequency data engineering pipelines in Python.
🏗️ The Architecture
The system is fully built in Python using asyncio. Here is the stack:
Data Ingestion: aiohttp WebSocket connection to Binance (@forceOrder and @kline_1m streams).
In-Memory Storage: Redis (via aioredis). I used Redis Sorted Sets (ZADD) to store timestamped trades and Kline data.
Processing Engine: A background worker that runs statistical analysis (σ-filters, CVD - Cumulative Volume Delta) every 60 seconds to detect anomalies.
💥 The Bottleneck: Event Loop Starvation
The bot ran perfectly for the first hour. But then, it started throwing TimeoutError exceptions when reading from Redis. The entire system would paralyze for 5-8 seconds.
At first, I thought I was exceeding the Redis connection pool limit. But the real culprit was Python's Asyncio Event Loop Starvation.
To calculate the Cumulative Volume Delta (CVD) and volatility multipliers, my background worker was fetching the last 4 hours of trades for all 100 coins using zrangebyscore.
For high-volume pairs like BTCUSDT or ETHUSDT, 4 hours of tick data is hundreds of thousands of JSON strings. When aioredis fetched this massive array from the socket, Python had to decode and parse it. Because Python's JSON parsing and list iteration are synchronous CPU-bound operations, it completely blocked the asyncio event loop.
While the CPU was busy parsing JSONs for 5 seconds, the WebSocket buffer filled up, and ping/pong keepalives failed, causing the timeouts.
🛠️ The Fix
I had to redesign the data processing logic without offloading to external microservices. Here is what I did:
- Architectural Data Reduction Instead of forcing the event loop to parse hours of data, I built a pre-processor that reduces the rolling window to just the minimum statistically significant timeframe.
By calculating the baseline volatility mathematically and extrapolating it, I reduced the payload size dropping from Redis by over 90%.
- Event Loop Micro-Yielding Even with optimized data payloads, iterating over thousands of tick events per second can be heavy. I implemented a micro-yielding pattern inside the synchronous loops:
python
Pseudo-code logic
for raw_trade in massive_data_stream:
if condition_met:
await asyncio.sleep(0) # Yielding control prevents starvation
process_complex_math(raw_trade)
(Also, swapping the standard json library for orjson in Python gave a massive parsing speed boost).
🚀 The Result
After deploying the fix, the CPU blocking disappeared entirely. The system now runs cleanly on a basic VPS, quietly maintaining rolling data windows for 100 pairs simultaneously. It successfully filters out 99% of market noise, allowing me to catch real algorithmic liquidations (like a massive $728K Stop-Hunt on DOGE that resulted in a perfect V-shape reversal—see images).
If you are building data pipelines in asyncio, always remember that network I/O isn't your only enemy—large payload parsing will silently kill your event loop!
P.S. A lot of folks ask if they can access these signals without building the pipeline from scratch. While I keep the core engine private, I did set up a platform where you can monitor these live liquidation sweeps and CVD data in real-time: https://sigma-liq.fun/index.html.
Feel free to drop by if you are curious about what the algorithmic data feed looks like in action. Let me know if you guys have any questions about handling high-frequency data in Python!
Top comments (0)