The Trading System Strikes Back: Dodging the Dark Side of Bugs

The Quest Begins (The "Why")

I still remember the first time I tried to turn a hobby project into a real‑time trading system. I was pumped, fueled by too much coffee and the dream of building something that could react to market moves faster than a human could blink. I wired up a simple WebSocket feed, threw together a quick order‑matching loop, and launched it on a testnet. For the first few minutes everything looked sweet—quotes streamed in, orders filled, and I felt like I’d just cracked the code.

Then the market got volatile. A sudden burst of trades hit, and my engine started dropping ticks like they were hot potatoes. Latency spiked, orders went stale, and I watched my simulated P&L swing wildly—not because of my strategy, but because the system couldn’t keep up. I spent hours staring at logs, wondering why a few extra messages could bring the whole thing to its knees. It felt like I was stuck in a boss fight where the enemy kept respawning faster than I could damage it.

That frustration sparked a question: What are the classic pitfalls that turn a promising trading prototype into a brittle, money‑leaking nightmare? I dug in, talked to a few quant friends, and ripped apart my own code. What I found were two sneaky traps that show up again and again, especially when we’re excited to ship fast.

The Revelation (The Insight)

The biggest revelation? Speed isn’t just about raw CPU cycles; it’s about predictable flow and precise math.

When we treat market data as a fire‑hose and just read it as fast as we can, we ignore back‑pressure. If the consumer can’t keep up, the producer overwhelms the queue, messages get dropped, and the system’s view of the world becomes stale. The fix isn’t to scream for more cores; it’s to design the pipeline so that speed is matched by bounded buffers and proper signalling.

The second insight hit me when I realized I was using Python’s float for price calculations. It seemed harmless—after all, a price like 100.25 is just a number, right? But floating‑point binary representation can’t exactly store most decimal fractions. Over thousands of operations those tiny rounding errors accumulate, turning a theoretically profitable spread into a loss or, worse, generating invalid order prices that get rejected by the exchange. The remedy? Switch to a decimal type that respects the exact tick size prescribed by the market.

Armed with those two lessons, I rebuilt the engine. The difference was night and day: no more dropped ticks, deterministic latency, and prices that stayed true to the exchange’s rules. It felt like finally seeing the boss’s pattern and landing the perfect combo.

Wielding the Power (Code & Examples)

Let’s look at the two concrete mistakes and how I turned them into strengths.

Trap #1 – Unbounded, Fire‑and‑Forget Data Ingestion

Before (the struggle)

import asyncio
import websockets

async def market_data_handler(ws):
    async for msg in ws:
        # Process each tick immediately – no buffering!
        process_tick(msg)          # <-- could be heavy, blocking
        # If process_tick lags, the ws queue grows without limit

async def main():
    async with websockets.connect("wss://test.exchange/feed") as ws:
        await market_data_handler(ws)

asyncio.run(main())

The problem? process_tick might take a few milliseconds under load. While it’s busy, the WebSocket keeps pushing new messages into the internal receive buffer. Eventually the buffer overflows, the connection drops, or we start silently discarding ticks—exactly what I saw during that volatility spike.

After (the victory)

import asyncio
import websockets
from asyncio import Queue

# A bounded queue protects us from sudden bursts
TICK_QUEUE: Queue = Queue(maxsize=10_000)

async def market_data_handler(ws):
    async for msg in ws:
        # If the queue is full we drop the oldest tick – better than crashing
        if TICK_QUEUE.full():
            _ = await TICK_QUEUE.get()   # discard
        await TICK_QUEUE.put(msg)

async def tick_worker():
    while True:
        tick = await TICK_QUEUE.get()
        await process_tick(tick)   # now we control the pace

async def main():
    async with websockets.connect("wss://test.exchange/feed") as ws:
        await asyncio.gather(
            market_data_handler(ws),
            tick_worker(),
        )

asyncio.run(main())

Now we have a producer‑consumer pattern with a fixed‑size queue. If the consumer falls behind, the producer gracefully drops the oldest tick instead of letting the socket buffer explode. The system stays connected, latency stays predictable, and we can monitor TICK_QUEUE.qsize() to know when we’re approaching capacity.

Trap #2 – Floating‑Point Price Math

Before (the struggle)

def calculate_spread(bid: float, ask: float) -> float:
    return ask - bid

# Example usage
bid_price = 100.123456
ask_price = 100.123470
spread = calculate_spread(bid_price, ask_price)
print(spread)   # 0.0000140000000001  <-- ugly rounding error

That tiny error might seem harmless, but when you multiply by position size or feed it into an order‑price calculator, you can end up submitting a price that violates the exchange’s minimum tick size (e.g., 0.00001). The exchange rejects the order, you miss the trade, and you waste precious latency on a round‑trip rejection.

After (the victory)

from decimal import Decimal, getlocalcontext

# Set precision high enough for the instrument’s tick size
getlocalcontext().prec = 10

def calculate_spread(bid: str, ask: str) -> Decimal:
    """Prices are passed as strings to avoid binary float contamination."""
    return Decimal(ask) - Decimal(bid)

# Example usage
bid_price = "100.123456"
ask_price = "100.123470"
spread = calculate_spread(bid_price, ask_price)
print(spread)   # 0.000014  -- exact, no surprise

By treating prices as immutable Decimal objects (or the equivalent in your language of choice), we guarantee that arithmetic respects the exact tick size defined by the market. The code is a tad more verbose, but the correctness pays off instantly when you see zero rejected orders due to price‑precision issues.

Why This New Power Matters

With these two patterns in place, your trading engine stops being a fragile house of cards and starts behaving like a well‑oiled machine:

• Deterministic latency – you know roughly how long a tick takes from arrival to decision, which lets you tune strategies with confidence.
• Zero data loss under bursts – the bounded queue acts as a shock absorber, protecting the connection even when the market goes nuts.
• Price integrity – every order price you send is exactly what the exchange expects, eliminating costly rejections and slippage from bad math.

The result? More filled orders, cleaner P&L curves, and the peace of mind to focus on what truly matters: refining your alpha, not firefighting infrastructure bugs.

Your Turn – The Challenge

Give these ideas a spin in your own codebase. Take a market‑data handler you’ve built, wrap it in a bounded async queue, and swap any float‑based price math for a Decimal (or fixed‑point) equivalent. Then run a short stress test with a replay of a volatile day and watch the difference in dropped ticks and order rejections.

What’s the biggest improvement you noticed? Did the latency graph flatten, or did your order‑reject rate plummet? Drop your findings in the comments—I’m eager to hear how your quest turned out!

---

May your queues stay bounded and your decimals stay precise. 🚀