DEV Community

EmilyL
EmilyL

Posted on

Reconstructing Historical Order Books with a Crypto API – A Step‑by‑Step Guide

#ai

We’ve all been there: you’re debugging a backtest, and the strategy made a terrible decision at 14:32:07. You check the candlestick—nothing unusual. You check the volume—normal. So what caused the slip? The answer almost always lies in the order book. But how do you get the order book as it was at that exact millisecond, long after the moment has passed?

In this tutorial, we’ll show you how to use a crypto API’s WebSocket stream to capture and store order‑book updates, so you can later retrieve a full snapshot for any given timestamp. We’ll share our own architecture, code snippets, and the lessons we’ve learned along the way.

Defining the Order‑Book Snapshot

A snapshot is a point‑in‑time representation of all active limit orders. It consists of:

  • Bids – buy orders with prices and quantities.
  • Asks – sell orders with prices and quantities.
  • Timestamp – when the snapshot was taken.
  • Symbol – the market pair.

Example (BTCUSDT):

{
  "symbol": "BTCUSDT",
  "timestamp": 1784188200000,
  "bids": [
    ["65000", "2.5"],
    ["64990", "1.8"]
  ],
  "asks": [
    ["65010", "1.2"],
    ["65020", "3.1"]
  ]
}
Enter fullscreen mode Exit fullscreen mode

With this data, we can analyse depth changes preceding price moves—essential for any serious quant.

The Problem: No Direct Historical Query

We scoured the documentation of every major crypto API. None provides a “get historical order book” method. The reason is obvious: order books mutate every few milliseconds, and storing all versions is prohibitively expensive. So we must design a system that subscribes to real‑time updates and persists them.

Comparing Data Collection Methods

Approach Accuracy Complexity Best For
HTTP REST polls Low (misses intra‑interval changes) Low Casual monitoring
WebSocket stream High (captures every event) Medium Backtesting, research

We chose WebSocket because our backtests require event‑level precision. Let’s see how to set it up.

Connecting to a Stream (Using AllTick API as an Example)

We used a provider that offers a straightforward WebSocket endpoint—we’ll call it AllTick API. The connection code is minimal:

import websocket
import json

url = "wss://apis.alltick.co/websocket-api/stock-websocket-interface-api/transaction-quote-subscription"

def on_message(ws, message):
    data = json.loads(message)
    print(data)

ws = websocket.WebSocketApp(
    url,
    on_message=on_message
)

ws.run_forever()
Enter fullscreen mode Exit fullscreen mode

In reality, we don’t just print. We parse the message, identify whether it’s a full snapshot or an incremental update, and then feed it into our storage pipeline. We also maintain a local in‑memory copy of the order book.

Storage Design: Choose Your Granularity

The volume of order‑book data is large. We recommend storing different densities for different purposes:

Purpose Storage Recommendation
Simple trend review Full snapshot every 5 seconds
Strategy backtesting Full snapshot every 200 ms + delta logs
High‑frequency research All deltas, no snapshots (reconstruct later)

We personally use a hybrid: we save full snapshots every second and archive all deltas. To reconstruct a specific time, we load the nearest snapshot and replay the deltas. Always use the exchange’s timestamp—not your local time—as the authoritative ordering key.

Critical Operational Checks

Over time, we’ve compiled this checklist to keep our data clean:

  • Timestamp gap detection – if the interval between two messages exceeds a threshold (say 100 ms), we assume packet loss and flag that range.
  • Reconnection logic – after a WebSocket disconnect, we first fetch a full snapshot (via REST) to re‑establish the baseline, then resume the delta stream.
  • Message type handling – we strictly separate snapshot (overwrite) from update (merge) to avoid double‑counting.
  • Sanity filters – we reject prices ≤0, quantities ≤0, and any value that deviates wildly from the current mid‑price.

These checks have drastically reduced our debugging time.

Why We Keep Building This Archive

We believe that the true power of a crypto API lies not in real‑time ticks, but in the historical depth it enables us to build. When we encounter a black‑swan event, we can roll back the tape and watch how the order book evolved—which levels were defended, which were abandoned. That insight informs our model adjustments and risk management.

We encourage every developer to start small—maybe just one pair, one snapshot per second—and expand over time. The data you accumulate will become your most valuable asset for strategy innovation.

Top comments (0)