DEV Community

EmilyL
EmilyL

Posted on

Handling Out‑of‑Order Level2 Messages from a US Stock API – A Practical Guide

#tutorial

As developers supporting quantitative trading desks, we’ve spent countless hours debugging order book inconsistencies. The root cause almost always traces back to one issue: message sequencing. In this post, we’ll share our battle‑tested approach to processing Level2 data from a US Stock API, focusing on how we maintain a reliable order book despite network‑induced reordering.

Client Requirements: Zero Tolerance for State Errors

Our primary users are professional traders and fund quant developers. They consume our Level2 feed to build real‑time liquidity models and execute automated strategies. Their non‑negotiable demand: the local order book must be a bit‑perfect replica of the exchange’s limit order book at any given moment. A single mis‑applied delta – for example, a modification applied before its corresponding add – can skew the entire price ladder and trigger incorrect trades. We learned this when a client reported a persistent spread miscalculation; after investigation, we found that out‑of‑order cancellation messages were the culprit.

The Pain Point: Network Jitter and Packet Reordering

Unlike consolidated tape data, Level2 streams consist of discrete order‑state transitions – insert, update, delete. These operations are inherently dependent. When you fetch Level2 via a typical US Stock API over WebSocket, you’re at the mercy of internet routing. It’s entirely possible for a later event to arrive before an earlier one. For instance, the exchange emits:
Insert (seq=100) → Update (seq=101) → Delete (seq=102)
But our receiver may see:
Update (101) → Insert (100) → Delete (102)
Applying them in arrival order would cause an “order not found” error on the update, and later an invalid insert. This becomes especially frequent during high‑volatility periods when message rates exceed 10,000 per second.

Sequence Numbers as the Single Source of Truth

We quickly dismissed timestamp‑based sorting – it’s unreliable due to clock skew and low resolution. Instead, we rely on the monotonic sequence field included in each Level2 message. Here’s an example payload:

{
 "symbol":"AAPL",
 "price":185.2,
 "volume":300,
 "sequence":10001
}
Enter fullscreen mode Exit fullscreen mode

We maintain a last_processed_seq variable. If the incoming sequence equals last_processed_seq + 1, we apply the delta. If it’s greater, we declare a gap and trigger a full snapshot recovery. If it’s smaller, we discard it as duplicate. This logic is simple yet effective – it catches missing messages immediately, preventing silent data corruption.

Architecture: Snapshot + Incremental with Recovery

Our production system follows a “snapshot + incremental” model:

  1. Initialization: Fetch a complete order book snapshot (all price levels and aggregated sizes) via a REST endpoint.
  2. Streaming: Open a WebSocket connection to receive incremental updates.
  3. Validation: For each delta, check sequence continuity.
  4. Recovery: On any gap, pause incremental processing, fetch a fresh snapshot, replace the local book, and reset the sequence counter.

We also implement a small delay buffer: when we receive a message with a sequence number slightly ahead (e.g., we expect 10003 but get 10004), we hold it for up to 50ms to see if 10003 arrives. If it does, we reorder and apply. If not, we fetch a snapshot. This reduces unnecessary full refreshes.

Code Walkthrough – Python WebSocket Listener

Below is the core implementation we use as a starting point. The example uses a common WebSocket endpoint (we’ve integrated with various providers; the pattern is identical). Notice the sequence validation in the callback:

import websocket
import json

last_sequence = 0

def on_message(ws, message):
    global last_sequence

    data = json.loads(message)

    seq = data.get("sequence")

    if seq and last_sequence:
        if seq != last_sequence + 1:
            print("Data gap detected – re‑sync required")

    last_sequence = seq

    print(
        data.get("symbol"),
        data.get("price")
    )

ws = websocket.WebSocketApp(
    "wss://api.alltick.co/ws",
    on_message=on_message
)

ws.run_forever()
Enter fullscreen mode Exit fullscreen mode

For production, we add:

  • Automatic reconnection with exponential backoff.
  • A thread‑safe cache for out‑of‑order messages.
  • Health checks that compare our local book against occasional snapshot hashes.
  • Metrics to monitor gap frequency and recovery latency.

Lessons Learned

The hardest part of Level2 processing isn’t writing the update logic – it’s ensuring the update order is correct. We’ve come to treat sequence validation as our primary defense against data corruption. If you’re building any system that consumes depth‑of‑market data, invest in this foundation first. Accurate prices follow accurate sequences – never the other way around.

Top comments (0)