We are a fintech engineering team. As technical leads, we build and operate real-time market data pipelines for quant developers and fintech companies. This article is a practical engineering note on how we handle latency and missing data when using forex API, stock API, and index API in intraday quant systems.
Scene: Three Markets, One Strategy Process
Our intraday strategies started with US equities only. Later we added forex and several major indices. The strategy process became cross-market. That change exposed a problem: forex, stocks, and indices do not share the same real-time behavior.
Forex trades across sessions. Stocks have official trading hours. Indices are derived from weighted constituents. If one polling loop tries to cover all three, the strategy will eventually misread the market.
Requirements: What Real-Time Data Must Provide
For our use case, the feed must provide:
- low and observable latency;
- recoverable reconnects;
- duplicate detection;
- per-asset-class isolation.
When forex API, stock API, and index API are subscribed together, these requirements are not optional. They determine whether a cross-market signal is trustworthy.
Data Pain: Latency Bottlenecks and Reconnect Gaps
Latency Is a Three-Stage Problem
Our first version used REST polling every few hundred milliseconds. It was simple, but during volatile periods the polling interval became the bottleneck. In forex, the European and US overlap can move fast. In stocks, pre-market and after-hours sparse prints can look stale. Index products have a different push cadence.
We split latency into network transport, server-side processing, and client-side consumption. Network transport depends on location. Server-side processing can be slowed by queued subscriptions. Client-side consumption depends on parsing, caching, and persistence. We then replaced high-frequency polling with WebSocket push and kept REST for K-lines and static data.
Missing Data After Disconnects
WebSocket reduced latency but introduced reconnect gaps. We compared three strategies:
| Approach | Implementation idea | What we observed |
|---|---|---|
| Simple reconnect, no backfill | Reconnect and resubscribe, discard the gap | Easiest to implement, but the missing window creates visible drift between backtest and live results, especially during forex night sessions |
| Reconnect plus REST backfill | Reconnect WebSocket and use REST to fetch recent ticks or K-lines | Covers most short disconnects with manageable complexity; this is our main production approach |
| Local queue plus server-side idempotent dedup | Client keeps a sequenced local queue; server messages carry unique IDs and duplicates are dropped | Best consistency, but higher development and maintenance cost; we only use it for high-frequency strategies with strict tick integrity requirements |
Most of our strategies use REST backfill after reconnect. The local queue plus idempotent dedup is reserved for strict tick-level workflows. After a forex API reconnected three times around a Non-Farm Payroll release, duplicate quotes affected our signal, so we added dedup where needed.
Solution: WebSocket Client with Heartbeat and Backfill
We also checked subscription fields against ALLTICK API’s public documentation when standardizing our config. The code below is a simplified version of our WebSocket client. The code block is kept as-is, with comments translated:
import websocket
import json
import time
import threading
# ========== Config ==========
TOKEN = "token" # replace with your actual token
WS_URL = f"wss://quote.alltick.co/quote-b-ws-api?token={TOKEN}"
# symbols to subscribe
SYMBOLS = ["BTCUSDT", "ETHUSDT"] # example
# ========== Callback functions ==========
def on_message(ws, message):
"""Receive and process pushed tick data"""
try:
data = json.loads(message)
cmd_id = data.get("cmd_id")
# 22998 is the tick data push protocol ID
if cmd_id == 22998:
tick = data.get("data", {})
print(f"Tick: {tick.get('code')} | "
f"Price: {tick.get('price')} | "
f"Volume: {tick.get('volume')} | "
f"Time: {tick.get('tick_time')}")
# do database write or strategy calculation here
else:
# print other responses (e.g. subscription confirmation 22005)
print("Response:", data)
except json.JSONDecodeError as e:
print("JSON:", e)
def on_error(ws, error):
print("WebSocket error:", error)
def on_close(ws, close_status_code, close_msg):
print("WebSocket closed")
def on_open(ws):
"""Send subscription request after connection"""
print("WebSocket connected, sending subscription...")
# build subscription request (protocol ID 22004)
subscribe_msg = {
"cmd_id": 22004,
"seq_id": 1, # custom, response will echo back
"trace": f"trace-{int(time.time()*1000)}", # must be unique per request
"data": {
"symbol_list": [{"code": symbol} for symbol in SYMBOLS]
}
}
ws.send(json.dumps(subscribe_msg))
print(f"Subscribed to: {SYMBOLS}")
# start heartbeat thread (send every 10 seconds)
def heartbeat():
while ws.sock and ws.sock.connected:
time.sleep(10)
try:
# send ping frame as heartbeat
ws.send("ping")
print("Heartbeat sent")
except Exception as e:
print("Heartbeat error:", e)
break
threading.Thread(target=heartbeat, daemon=True).start()
# ========== Main program ==========
if __name__ == "__main__":
ws = websocket.WebSocketApp(
WS_URL,
on_open=on_open,
on_message=on_message,
on_error=on_error,
on_close=on_close
)
# recommend adding auto-reconnect logic
while True:
try:
ws.run_forever()
print("Reconnecting in 3 seconds...")
time.sleep(3)
except KeyboardInterrupt:
print("Exiting...")
break
The heartbeat thread is not decoration. Without it, quiet periods can trigger server-side disconnects. Combined with REST backfill and, when necessary, idempotent dedup, the feed becomes observable and recoverable.
For intraday quant, forex API, stock API, and index API should be treated as separate data contracts. Latency and missing data are architecture concerns, not after-launch fixes.
Top comments (0)