DEV Community: James Tao

My Production Pipeline for 24/7 Level 2 Market Data Collection

James Tao — Tue, 21 Jul 2026 02:53:52 +0000

Intro

If you’re a quant dev, algorithmic trader, or fund research engineer, you’ve almost certainly run into this frustrating issue: models trained only on tick trades and Level 1 top-of-book data perform great in backtesting, yet fall apart the second you run live simulation. Critical order flow signals disappear entirely.
I’ve spent years building market data pipelines for institutional quantitative teams. When we built an intraday volatility forecasting system for an equity fund, our initial data stack only pulled trade prints and single-tier quotes. It worked fine during quiet sideways markets, but signals lost all accuracy around opening auctions, closing volume spikes, and large block order events.

After deep debugging, we found the core limitation: tick data only records completed trades, and Level 1 data hides layered limit orders stacked far from the current price. Without full market depth, you can’t map supply and demand shifts ahead of price moves.
This guide walks through a production-ready WebSocket Level 2 data pipeline, including data comparison, connection architecture, local order book logic, production stability fixes, and minimal Python code you can drop into your project.

1. Tick vs Level 1 vs Level 2: When to Use Each Feed

Before building any trading model, clarify the purpose of each market data source — picking the wrong feed ruins your analytical depth.
Tick-by-Tick Trade Data
Logs every matched order’s price and volume. Great for candlestick generation and historical backtesting. The downside: it only shows past trades, no visibility into pending limit orders that will drive future price action.
Level 1 Top-of-Book Data
Only returns the best bid, best ask, and their respective volumes. Perfect for simple market dashboards, but unable to detect hidden buy/sell walls across distant price levels. You’ll consistently underestimate support and resistance strength.
Level 2 Full Depth Data
Stores aggregated resting order volume for every price tier on both sides of the market. This is the foundational feed for measuring real-time liquidity and spotting early order imbalance signals before prices shift.
Real-world example: Large buy orders stacking at key support, or mass sell order cancellations at resistance — these predictive patterns only appear in Level 2 streams, never in finished tick data.

2. Why WebSockets Beat HTTP Polling for Level 2 Data

Traditional polling has fatal flaws for high-frequency order book data: fixed-interval requests skip millisecond book updates, and repeated API calls waste server bandwidth and inflate long-term cloud costs.
Level 2 order books refresh constantly, so bidirectional persistent WebSocket connections are the industry standard. The core workflow has four simple steps:
Establish a permanent WebSocket tunnel between your client and market data gateway
Send a subscription request with your target tickers and data type (Level 2)
Receive continuous incremental delta updates whenever the order book changes
Refresh your local order book snapshot using cached historical state
A common beginner mistake: most data APIs send only delta changes, not full resets every message. If you don’t cache the latest full book locally, delta-only messages will create missing price tiers and miscalculated total order volume, breaking all your liquidity metrics.

3. Core Level 2 Fields & Local Order Book Storage Logic

Every incremental Level 2 message relies on five mandatory fields we standardize across all production pipelines:
symbol: Ticker identifier to separate data for multiple instruments
side: Marks bid (buy) or ask (sell) order layers
price: The price tier being updated
size: Remaining resting order quantity at this price
timestamp: Event time for cross-data time alignment and outage validation
Our production design splits buy and sell layers into two independent arrays.
When we receive an update:
Adjust the volume value for the matching price tier based on side
Delete the tier entry if size drops to zero
This setup removes full array scans when calculating top bid/ask or total market depth, cutting down compute latency and supporting parallel monitoring of dozens of tickers at once.

4. Minimal Working Python WebSocket Snippet

We separate network transport and data parsing logic for stable cloud deployments. Our dev environments use AllTick API as the market data provider. This stripped-down example works out of the box; you just add tier parsing logic for full production order book maintenance:

import websocket

def receive_data(ws, msg):
    data = eval(msg)
    code = data.get("symbol")
    price = data.get("price")
    vol = data.get("volume")
    print(f"Ticker: {code}, Price: {price}, Order Volume: {vol}")

def connect_init(ws):
    sub_info = {"action": "subscribe", "symbol": "MSFT", "type": "level2"}
    ws.send(str(sub_info))

if __name__ == "__main__":
    ws_client = websocket.WebSocketApp("wss://api.alltick.co/stock/websocket",
                                       on_open=connect_init,
                                       on_message=receive_data)
    ws_client.run_forever()

5. Four Production Stability Improvements for 24/7 Pipelines

Level 2 streams generate thousands of delta messages per second. Network drops and timezone mismatches easily break data continuity. After deploying dozens of institutional pipelines, we standardized four fail-safe workflows to eliminate manual data repair work:
Scheduled full order book snapshot caching
Save a complete copy of all bid/ask tiers on a fixed interval. After disconnection, restore the full book instantly from the snapshot instead of requesting full historical resyncs.
Timestamp sequence validation filter
Compare incoming message timestamps against local cache records. Auto-discard out-of-order and duplicate messages to prevent duplicate volume counting errors.
Auto resync after WebSocket disconnect
Once the tunnel reopens, trigger supplementary data requests to fill missing delta records from the outage window.
Abnormal price/volume threshold filtering
Set reasonable min/max bounds for price swings and order sizes. Auto-filter noise like flash price jumps and spoof oversized orders to avoid skewing your quant indicators.
Pro tip: Most market gateways output UTC timestamps. If you mix UTC with local time during backtesting or time-window aggregation, your entire timeline will misalign. Normalize all timestamps during initial parsing to avoid this common bug.

Wrap Up

A lot of new quant engineers treat WebSocket connection setup as the entire Level 2 integration process. In reality, data reliability hinges on underrated engineering details: persistent snapshot storage, automatic outage recovery, and unified time synchronization.

Tick data only shows completed trades, while full Level 2 depth exposes pre-trade order flow that drives price moves. Building a standardized Level 2 pipeline adds multi-layer analysis for intraday forecasting and liquidity risk tracking. For teams monitoring short-term capital shifts, this stack drastically reduces performance gaps between backtesting and live trading, delivering far more reliable trading signals.

What Data Streams Do You Need to Build a US Stock Order Imbalance Market-Making Strategy?

James Tao — Mon, 20 Jul 2026 02:56:56 +0000

Intro

I’ve been building automated market-making bots for US equities as a side quantitative project, and I ran into a frustrating, widespread pain point most new algo traders hit:
My backtest results looked rock-solid on local historical data, but once I deployed the code to cloud servers for paper trading, quoting logic went haywire. Risk metrics spiked continuously, and bid/ask spreads drifted far away from reasonable market levels.
I spent hours auditing my imbalance formulas and quote adjustment logic, only to find zero bugs in the core algorithm. The real root cause was incomplete, unsynchronized market data pipelines.

In this post, I’ll break down the full suite of data streams required for reliable order book imbalance strategies, walk through common data architecture pitfalls, share a standardized cloud processing workflow, and include a minimal WebSocket subscription code snippet using AllTick API for real-time US market feeds. This guide is tailored for self-taught quant devs, algo engineers, and anyone building micro-structure trading systems.

Core Data Standards for Order Imbalance Market-Making

Order imbalance strategies dynamically adjust bid/ask quotes based on real-time supply-demand shifts in the limit order book. US stock liquidity behaves drastically differently across pre-market open, regular session, and closing auction windows, so your data pipeline must satisfy four non-negotiable requirements:
Full Level 2 order book depth coverage
Single-level bid/ask snapshots are useless for calculating accurate imbalance ratios. You need complete volume aggregates for every price tier on both sides of the book.
Raw tick-by-tick trade stream integration
Order books only display resting liquidity intent; executed tick data reveals genuine capital buying/selling pressure hidden behind temporary spoofed orders.
Unified timestamp synchronization across all feeds
Mismatched timestamps between book snapshots and trades break imbalance calculations entirely, creating false micro-structure signals.
Archived historical market data for multi-scenario validation
You need historical tick and minute bar archives to test strategy performance under low-liquidity, high-volatility, and opening/closing auction market regimes to avoid overfitting.

Missing or delayed data of any type creates massive divergence between backtest curves and live/paper trading performance.

Common Data Pipeline Pitfalls for New Quant Developers

After chatting with dozens of self-taught algo builders, these four architectural mistakes consistently derail imbalance strategy deployments:
Only consuming top-of-book data, missing full Level 2 depth
This skews your core imbalance metric, as hidden large orders on distant price tiers are completely ignored.
Subscribing to order book feeds without tick trade streams
You cannot distinguish transient fake orders from sustained institutional buying/selling activity.

Separate, unsynchronized ingestion for book, trade, and volume data
No centralized timestamp alignment leads out-of-order data records that corrupt real-time factor computations.
No persistent historical data archiving
You can only test your strategy on narrow market windows and cannot validate robustness across full market cycles.
Most developers sink dozens of hours tweaking mathematical formulas while ignoring foundational data infrastructure—this is why so many algos fail when moved out of backtesting environments.

Standard Cloud Data Preprocessing Pipeline (24/7 Paper Trading Ready)

This universal workflow works for long-running cloud-based quant systems:

Establish persistent WebSocket connections to ingest raw market data into volatile in-memory cloud cache
Automated cleaning layer filters out outlier price jumps and duplicate duplicate broadcast records
Cross-align all book, tick, and volume streams using the unified timestamp standard
Normalize all dataset field schemas, then split the pipeline into two branches: real-time computation and historical archiving
Real-time branch feeds the imbalance market-making strategy module for live quote generation; archived branch writes time-series storage for batch backtesting

Minimal WebSocket Subscription Demo Code

This base snippet establishes a live US equity tick feed subscription. Production deployments require additional modules for timestamp validation, persistent Redis caching, automatic reconnection, and time-series database writes.

import websocket
import json

# US stock real-time trade subscription payload
subscription_payload = {
    "type": "transaction_quote",
    "symbol": "market.usstock"
}

def on_open(ws):
    ws.send(json.dumps(subscription_payload))

def on_message(ws, raw_message):
    market_data = json.loads(raw_message)
    # Extend here: cache writes, timestamp alignment logic
    print(market_data)

if __name__ == "__main__":
    ws_client = websocket.WebSocketApp(
        "wss://quote.alltick.co/websocket-api",
        on_open=on_open,
        on_message=on_message
    )
    ws_client.run_forever()

Final Thoughts

Complex mathematical imbalance formulas and quote adjustment logic are only the surface layer of a viable market-making strategy. The backbone of consistent, reliable paper/live trading performance is a complete, time-synchronized low-latency data pipeline.

Combining Level 2 depth, tick execution archives, historical market records, aggregated volume metrics, and cross-feed timestamp normalization resolves nearly all common discrepancies between backtesting and live algorithm behavior.
Prioritizing robust data infrastructure over endless formula iteration drastically shrinks performance gaps and makes your strategy’s output far more trustworthy for real capital deployment.

Why Do US Stock Minute Bar Backtests Fail to Match Live Trading Results?

James Tao — Fri, 10 Jul 2026 02:52:04 +0000

Intro

If you’ve built intraday quantitative strategies for US equities, you’ve almost certainly hit this frustrating roadblock: a strategy that delivers strong returns on historical minute bars completely falls apart when run against live real-time market data. Most developers waste hours debugging indicator logic and tuning parameters, only to find the root cause lies not in strategy code, but inconsistent underlying market data rules between historical compressed bars and raw live tick streams.

US stock markets add extra complexity here with multiple time zones, pre-market/after-hours sessions, and corporate action adjustments, all of which widen the gap between backtest simulation and live execution. In this article, I’ll break down four critical data flaws, and cover a modular project structure for long-term quantitative research.

Four Core Data Issues That Break Backtest-Live Consistency

1. Aggregated minute bars discard granular intraday tick movement
Minute bars are compressed aggregates of all ticks within a single trading window, only storing open, high, low, close prices and total volume. All rapid price swings, intraday reversals, and micro inflection points are permanently lost during aggregation.
For short-term momentum and reversal strategies that rely on split-second price shifts, historical minute bars only show the final state of each minute. Live tick streams capture every individual trade, which creates entirely different entry/exit trigger conditions. This mismatch leads to inconsistent trade counts and drastically different profit outcomes.

2. Conflicting time standards create timeline misalignment
US market data uses three separate time references: exchange native time, UTC, and server local time. Mixing these standards is a common oversight that introduces persistent timeline offset.
If backtest logic calculates indicators using exchange timestamps while live code references local server time, your rolling data window will constantly shift. Even minor time offsets disrupt moving average calculations, volume aggregation, and signal triggering, compounding performance divergence over extended runs.
My standard workflow normalizes all timestamps to exchange time upfront and labels the exact trading range of every bar to eliminate timeline errors at the source.

3. Mismatched price adjustment & outlier filtering logic
Historical minute data is almost always adjusted for stock splits and dividends to eliminate price gaps, while live streaming data returns raw unadjusted market prices. Any price-based metrics and profit calculations will carry systematic bias if backtest and live environments use different price rules.
On top of that, every data provider implements unique filtering logic for flash orders, empty zero-volume bars, and sparse pre/post-market trades. These small inconsistencies accumulate and further separate backtest performance from live results.
When evaluating market data APIs, I always verify four key criteria: unified exchange timestamps, transparent tick-to-minute aggregation rules, toggleable corporate action adjustments, and uninterrupted full tick streaming.

4. Static precomputed bars cannot replicate live streaming behavior
Ready-made historical minute bars are loaded as static bulk datasets all at once. Live trading consumes ticks incrementally via persistent WebSocket streams. This fundamental difference in data delivery creates an unavoidable environmental divide between backtesting and live deployment.

Standardized Fix: Generate Minute Bars From Raw Live Ticks
The most reliable way to eliminate backtest-live divergence is to align the entire data generation pipeline for both historical simulation and live execution, instead of relying on pre-built static minute bars.
The industry-standard approach I use for US equity research reconstructs minute bars directly from raw tick data, mirroring the exact aggregation logic used in live production.

Minimal Python Tick Subscription Snippet

import websocket
import json

ws_endpoint = "wss://quote.alltick.co/quote-b-ws-api"

def on_tick_received(ws, raw_payload):
    tick_data = json.loads(raw_payload)
    # Implement custom tick caching & minute bar aggregation here
    print("Incoming real-time tick data:", tick_data)

if __name__ == "__main__":
    ws_client = websocket.WebSocketApp(ws_endpoint, on_message=on_tick_received)
    ws_client.run_forever()

Building minute bars from continuous raw ticks fully replicates real-time volatility patterns and recovers the intraday detail lost in pre-aggregated K-lines, making backtest performance far more representative of live trading reality.

Scalable Modular Architecture For Expanded Quantitative Projects
If you plan to add persistent data storage, batch backtesting pipelines, or real-time price alerting, a decoupled three-layer architecture simplifies collaborative development and future iteration:
Preprocessing Layer: Standardize timestamps, toggle split/dividend adjustments, filter abnormal outlier trades
Market Aggregation Layer: Cache raw tick data and roll standardized minute bars using exchange time
Strategy Execution Layer: Reuse identical K-line parsing and indicator calculation functions for both backtesting and live simulation
Modular separation lets you add new asset classes or timeframes without breaking core data calibration logic.

Wrap-Up
Most quantitative developers instinctively assume flawed strategy logic is to blame when backtest results fail to translate live. Consistent testing proves that standardized timestamps, uniform minute bar generation rules, and matching price adjustment handling are the primary levers that determine backtest credibility.

US equities’ multi-timezone structure, extended trading hours, and corporate action rules create natural separation between static historical minute bars and dynamic live tick streams. By unifying tick ingestion, normalizing timestamps, and locking consistent price processing rules, your backtest curves gain actionable real-world relevance and make it far easier to isolate genuine strategy performance drivers during optimization.

How to Stream A-Share Stocks + ETFs Simultaneously With One Python WebSocket Connection

James Tao — Thu, 09 Jul 2026 02:42:00 +0000

## Intro
If you build quantitative monitoring scripts or real-time trading data pipelines, you’ve definitely faced this common ask: ingest real-time tick data for individual A-share stocks and ETFs at the same time. Traders usually combine single-stock data (tracking individual corporate capital flow) with ETF data (gauging sector/broad market sentiment) to get a full view of intraday momentum.

A lot of new devs fall into two inefficient anti-patterns: spinning up two separate WebSocket connections, or polling APIs on a fixed interval. Both approaches waste cloud resources, bloat your codebase, and introduce data gaps during volatile price swings. In this post, I’ll walk through a clean, scalable single-stream architecture, share runnable Python code, and cover production tweaks for long-running cloud deployments.

Common Pitfalls With Separate Data Pipelines

1. Dual WebSocket Connections Waste Bandwidth & Maintenance
Running independent WebSocket clients for stocks and ETFs doubles your network overhead and connection pool usage. Every piece of error handling, reconnection logic has to be duplicated. Later, if you want to add indices or convertible bonds, you’ll copy-paste huge blocks of code and create a messy, hard-to-debug repo.

2. Periodic Polling Creates Missing Tick Data
HTTP polling can’t capture rapid intraday price spikes or flash orders. When you miss ticks, your backtest datasets become distorted, and any live trading signals generated from that data lose reliability. Polling is never a fit for 24/7 real-time quant monitoring.

3. Hardcoded Tickers Hurt Iteration Speed
Embedding stock/ETF symbols directly inside your connection logic means you have to edit core network code every time you update your watchlist. This introduces unnecessary risk of breaking the streaming client during research iteration.

Single Unified Stream Architecture Breakdown

Stock and ETF tick payloads share nearly all core fields: real-time price, volume, order book levels, and timestamps. The only difference is asset type classification, which we can handle post-ingestion instead of splitting the stream upfront.

My standard workflow rules:

Maintain one central watchlist storing both stock and ETF tickers;
Open a single persistent WebSocket connection and batch-subscribe all instruments at once;
Parse incoming unified tick data, then branch business logic for stocks vs ETFs locally;
Decouple watchlist config from low-level WebSocket logic so ticker changes never touch connection code.

Minimal Working Python Code

import websocket
import json

# Unified watchlist with A-share stock + ETF codes
watchlist = ["600000", "510300"]

def on_open(ws):
    # Send bulk subscription request once connection establishes
    sub_payload = {
        "cmd": "subscribe",
        "symbol_list": watchlist
    }
    ws.send(json.dumps(sub_payload))

def on_message(ws, raw_msg):
    tick = json.loads(raw_msg)
    ticker_code = tick.get("symbol")
    live_price = tick.get("price")
    ts = tick.get("timestamp")
    # Insert custom stock/ETF separate processing logic here
    print(f"Ticker: {ticker_code} | Price: {live_price} | Timestamp: {ts}")

if __name__ == "__main__":
    ws_endpoint = "wss://quote.alltick.co/quote-b-ws-api"
    ws_client = websocket.WebSocketApp(ws_endpoint, on_open=on_open, on_message=on_message)
    print("Streaming live data for stocks + ETFs in a single connection")
    ws_client.run_forever()

Production Optimizations For 24/7 Cloud Uptime

This minimal snippet works for local testing, but you need these four tweaks for stable long-running deployments:

Normalize ticker formatting: Different data providers add market prefixes (SH/SZ) inconsistently. Standardize all codes before writing to time-series DBs to avoid matching errors during backtesting.
Unify timestamp timezone: Convert all raw millisecond/nanosecond timestamps to Beijing Time to eliminate misalignment when building minute K-lines or cross-asset momentum calculations.
Auto-reconnect + resubscribe logic: Cache your full watchlist in memory. If the WebSocket drops from network blips or rate limits, automatically re-subscribe all instruments after reconnecting without manual restarts.
Data freshness filtering: Low-liquid ETFs and small-cap stocks can sit quiet for minutes. Add a time threshold to deprioritize outdated tickers in dashboards so stale data doesn’t skew market rankings.

Wrap Up

Building separate streaming pipelines for stocks and ETFs creates unnecessary engineering overhead. A single WebSocket connection paired with a unified watchlist reduces resource usage, simplifies maintenance, and lets you extend coverage to other asset types with minimal code changes.

The core pillars of reliable continuous ingestion are standardized tick formatting, consistent timestamps, and automatic reconnection handling — these small tweaks separate throwaway test scripts from production-grade quant data pipelines.

Build a Stable Pre/Post Market US Stock Ranking Dashboard

James Tao — Wed, 08 Jul 2026 03:11:42 +0000

**Intro
When building market data pipelines for US stock quantitative research, most developers start by only checking if an API can return real-time tick data. After weeks of unattended runtime testing, we quickly learn the real pain point: inconsistent data handling during pre-market and after-hours extended trading sessions.

If your gain/loss ranking dashboard shifts stock positions randomly every few seconds, all your backtesting datasets and real-time strategy signals become unreliable. This ranking system isn’t just a simple fetch-and-render frontend project — the core engineering work lies in normalizing timestamp logic, baseline pricing rules, and sorting logic for low-liquidity extended hours market data.

Better Baseline Logic for Extended Hours Pct Change
The most common rookie mistake is using the previous day’s closing price as the single baseline for all trading windows. This artificially inflates volatility readings in pre and post market, skewing your entire leaderboard.

I use segmented baseline pricing to eliminate this bias:
Pre-market: Set baseline as the first stable matched trade price before market open
After-hours: Set baseline as the first valid trade price once regular hours close
Never reuse the single previous close price across all sessions
For ranking pipelines, consistent calculation rules beat ultra-fast recalculation every time. If your baseline logic shifts between ticks, stock positions will flicker endlessly on the dashboard.

3-Tier Decoupled Data Architecture (Avoid Frontend Heavy Computation)
I split the data flow into three isolated layers to insulate the UI from network lag, bad ticks, and calculation spikes:

Raw Tick Layer: Persist all original market ticks for backtesting replay and error tracing
Central Calculation Layer: Handle unified pct change math and global sorting logic
Presentation Layer: Render pre-computed ranking results with zero live arithmetic Add a short sliding tick aggregation window as an extra stabilization step. Group all ticks within a 2–5 second window before sending batches to the calculation layer — this smooths out constant micro price jumps that cause leaderboard chaos.

Minimal Python Implementation: Baseline Tracking & Pct Calculation

import websocket
import json
from collections import defaultdict

# Store session baseline price per ticker
base_price_map = defaultdict(float)

def on_message(ws, raw_message):
    data = json.loads(raw_message)
    symbol = data.get("symbol")
    price = float(data.get("price", 0))
    timestamp = data.get("timestamp")

    # Lock initial baseline price on first tick of the session
    if base_price_map[symbol] == 0:
        base_price_map[symbol] = price
    base_price = base_price_map[symbol]
    change_pct = (price - base_price) / base_price * 100 if base_price else 0
    print(f"Ticker: {symbol} | Price: {price} | 24h Change: {round(change_pct, 2)}%")

if __name__ == "__main__":
    ws_client = websocket.WebSocketApp(
        "wss://api.alltick.co/ws",
        on_message=on_message
    )
    ws_client.run_forever()

Fix Stale Data Bias (A Frequently Overlooked Edge Case)
Ticker update frequency varies wildly during extended hours: high-volume large-cap stocks stream ticks nonstop, while low-liquidity small caps may sit dormant for minutes at a time. Without filtering, outdated old prices will falsely push inactive stocks to the top of your leaderboard.

A simple lightweight filter fixes this: if a ticker has no new tick data within your defined freshness threshold, lower its sort priority or temporarily remove it from the ranking list entirely. This small tweak drastically improves the realism and trustworthiness of your real-time dashboard.

Final Engineering Takeaways
When building US stock market data infrastructure, raw network latency of your API feed is not the biggest factor impacting dashboard quality. The single most impactful variable is consistent, session-separated data processing for pre-market and after-hours ticks.

Nearly all flickering, unreliable leaderboards stem from mixing tick data from disjoint trading windows under one universal calculation formula. Three simple adjustments resolve almost all ranking instability: separate session classification logic, segmented baseline pricing rules, and throttled refresh cadences.

Once these standards are built into your pipeline, your ranking dashboard will reflect genuine market movement instead of a chaotic, constantly shifting list of numbers.