Intro: A Tricky Production Bug I Hit While Building Quant Market Terminals
While building real-time market terminals for quantitative teams, I ran into a subtle but damaging production issue. After stocks exit long trading halts, candlestick charts show blank time gaps, and all backtesting metrics (returns, volatility, risk ratios) become unreliable for strategy testing.
My first naive solution made everything worse. Whenever Tick data stopped arriving, I assumed the WebSocket dropped, spammed reconnection attempts, and continuously pulled historical K-line data to fill gaps. After launch, API traffic tripled, bandwidth costs shot up, duplicate snapshot records flooded the database, and data validation errors popped up constantly.
After a week of debugging, I built a reliable standardized pipeline. It uses standard WebSocket subscription commands and parses built-in trade status tags inside payloads. The system manages multiple instruments over one persistent connection, avoids repeated connection creation, and never generates fake trade data for halted sessions. It automatically reconstructs continuous timelines covering halt periods and post-resumption live feeds.
In this post, I’ll walk through core logic, four common production pitfalls, and fully runnable Python code you can drop straight into your financial data project.
Core Concept: Snapshot Recovery for Halt & Resumption Cycles
Snapshot recovery triggers the second the first post-resumption Tick arrives. The system reads locally cached halt metadata, validates historical snapshot integrity across the suspended window, and inserts only trade status markers to link pre-halt static price snapshots with live post-resumption streams.
This approach beats two inefficient common implementations:
- No tearing down/rebuilding WebSocket connections, which prevents reconnection storms from massive short-lived sockets.
- No scheduled REST polling for full historical K-line batches, cutting unnecessary API load.
Use Cases & Validation Reference Table
| Use Case | Main Dev Pain Point | Subscription Config (cmd_id/action/code) | Validation Standard |
|---|---|---|---|
| Initial subscription with halted stocks | Cannot tell API disconnection apart from exchange suspension, triggers unnecessary backfill requests | cmd_id=22004, action=subscribe, parse status field |
Cache each instrument’s trading state right after WebSocket handshake |
| Intraday temporary halt | Brief Tick silence triggers frequent reconnections, stacking short server connections | Keep one persistent connection, detect suspend flag in payload |
Log halt windows locally and save the last pre-halt price |
| Long halt followed by resumption | Raw resumption Ticks write directly to DB without timeline context, creating permanent chart gaps | Reuse existing WebSocket channel and trigger snapshot repair logic | Cross-check halt start/end timestamps and fill missing timeline records |
| Duplicate subscribe requests for halted stocks | Repeated upstream commands create redundant snapshot rows in storage | Deduplicate instrument codes via local cache before sending requests | Block redundant subscribe commands if the instrument exists in cache |
| Network outage overlapping stock resumption | Halt state records vanish after reconnect, breaking continuity between old and new market data | Auto-restore subscriptions post-reconnect, batch fetch instrument baseline status | Run full historical snapshot integrity check once reconnection finishes |
Four Critical Production Pitfalls for Market Data Developers
1. Treat halted Tick silence as full data loss, loop historical API requests endlessly
Symptom: No trade payloads appear during suspension, yet the service keeps calling batch K-line endpoints, burning through API quota limits quickly.
How to detect: Parse the status field in every Tick payload to separate two silent window types: exchange suspension vs real network disconnection.
Fix rule: Only trigger reconnection and backfill when status=normal paired with prolonged data silence. Disable all backfill logic while instruments are suspended.
2. Save resumption Ticks without linking pre-halt metadata, breaking timeline continuity
Symptom: The database only stores the last snapshot pre-halt and the first Tick post-resumption. No intermediate state records exist, leaving visible blank gaps on charts.
How to detect: Pull the full time-series log for a single stock and check continuity between halt end timestamps and resumption Tick timestamps.
Fix rule: Create a dedicated metadata table storing halt start/end timestamps and pre-halt closing prices. Attach these records to every resumption Tick write to fill timeline gaps with state markers.
3. Parallel repair workflows for multiple resumed stocks create database write race conditions
Symptom: Multiple stocks resume trading on the same day; multi-threaded parallel repair generates duplicate halt metadata entries for identical stock codes and suspension windows.
How to detect: Count records grouped by instrument code + halt start timestamp — duplicate entries confirm race conditions.
Fix rule: Serialize snapshot repair logic per instrument. Add a composite unique database index on code + suspend_start to block duplicate storage.
4. Mix asset-class WebSocket endpoints, making halt status fields unparseable
Symptom: Equity stocks subscribed via crypto WebSocket endpoints receive payloads missing the status field, making halt/resumption detection impossible.
How to detect: Audit connection domains — equities require a dedicated isolated WebSocket address.
Fix rule: Enforce asset-class routing logic in code. Route all stock requests to the exclusive equity WSS endpoint and block misconfigured cross-asset requests.
Solution Scope & Limitations
This pipeline is built around the standard subscription command cmd_id=22004. It supports dynamic instrument add/remove and halt timeline reconstruction within a single active WebSocket connection. Two hard limitations to note before implementation:
- Trading halt states cannot sync across multiple independent WebSocket connections.
- The system will not auto-generate synthetic Tick records for suspended periods; only descriptive status markers are appended, no fabricated market data.
Full Runnable Python Code
import websockets
import asyncio
import json
from datetime import datetime
# Dedicated stock WebSocket endpoint, referenced from official API docs
WSS_STOCK_URL = "wss://quote.alltick.co/quote-stock-b-ws-api?token=YOUR_TOKEN"
class StockQuoteClient:
def __init__(self):
self.ws = None
self.subscriptions = set()
# Local cache: key = stock code, value = halt timestamps, pre-halt price, trade state
self.stock_status_cache = {}
async def send_subscribe(self, action: str, code_list: list):
if not code_list:
return
payload = {
"cmd_id": 22004,
"action": action,
"code": code_list
}
await self.ws.send(json.dumps(payload))
if action == "subscribe":
[self.subscriptions.add(c) for c in code_list]
elif action == "unsubscribe":
[self.subscriptions.discard(c) for c in code_list]
def check_resume_repair(self, code: str, curr_status: str, trade_time: str):
"""Core logic: Detect stock resumption and trigger timeline snapshot repair"""
cache_info = self.stock_status_cache.get(code)
if not cache_info:
return
old_status = cache_info["status"]
# Trigger repair when state switches from suspended to normal trading
if old_status == "suspend" and curr_status == "normal":
print(f"Instrument {code} resumed trading; running historical snapshot timeline validation")
self.repair_snapshot_timeline(code, cache_info["suspend_start"], trade_time)
self.stock_status_cache[code]["status"] = "normal"
def repair_snapshot_timeline(self, code, suspend_start, resume_time):
"""Simulate timeline reconstruction & persist halt-period metadata records"""
repair_record = {
"code": code,
"suspend_start": suspend_start,
"resume_time": resume_time,
"pre_suspend_price": self.stock_status_cache[code]["last_price"],
"status": "suspend_repaired"
}
# Replace with your production storage logic: save_market_snapshot(repair_record)
print("Saved timeline repair record for halt window", repair_record)
async def on_open(self):
# Initialize watchlist instruments
init_codes = ["NASDAQ:AAPL", "HKEX:00700"]
await self.send_subscribe("subscribe", init_codes)
print("Stock WebSocket connection opened, initial instrument subscription finished")
async def on_message(self, raw_msg):
if not raw_msg:
return
try:
data = json.loads(raw_msg)
tick_data = data.get("data", {})
code = tick_data.get("code")
price = tick_data.get("price")
trade_time = tick_data.get("time")
status = tick_data.get("status", "normal")
# Guard clauses to filter invalid empty payloads
if not code or price in (None, 0) or not trade_time:
return
# Update local instrument state cache
if code not in self.stock_status_cache:
self.stock_status_cache[code] = {}
self.stock_status_cache[code]["last_price"] = price
self.stock_status_cache[code]["status"] = status
if status == "suspend" and "suspend_start" not in self.stock_status_cache[code]:
self.stock_status_cache[code]["suspend_start"] = trade_time
# Check if resumption repair workflow needs to run
self.check_resume_repair(code, status, trade_time)
print(f"Market update | {code} Price: {price} Trading State: {status}")
except Exception as e:
print("Failed to parse market Tick payload", str(e))
async def on_error(self, err):
print("WebSocket connection error:", err)
async def on_close(self):
print("Stock WebSocket connection closed")
async def connect(self):
try:
async with websockets.connect(
WSS_STOCK_URL,
ping_interval=10
) as ws:
self.ws = ws
await self.on_open()
while True:
msg = await ws.recv()
await self.on_message(msg)
except Exception as e:
await self.on_error(e)
await self.on_close()
async def run_demo():
client = StockQuoteClient()
task = asyncio.create_task(client.connect())
await task
if __name__ == "__main__":
asyncio.run(run_demo())
Wrap Up
Most market data engineers focus only on real-time price streaming and overlook edge cases like trading halts and resumptions — issues that only break things once quant backtesting or client charting goes live. A robust snapshot recovery stack relies on three coordinated layers: real-time stream state monitoring, local instrument metadata caching, and historical snapshot timeline validation. Combined, these eliminate candlestick gaps and skewed calculation metrics before they reach end users.
If you don’t want to build full multi-asset market data infrastructure from scratch and need to implement this halt-resumption repair workflow quickly, AllTick API simplifies the process. Its standardized WebSocket subscription schemas and multi-language sample code drastically cut engineering overhead for special trading state logic across stocks, forex, precious metals, and crypto.

Top comments (0)