Optimize US Stock Data Fetch: WebSocket Dynamic Subscription with Python

When building quantitative trading strategies and running backtests for US stocks with Python, slow market data retrieval is a common pain point I’ve run into many times.

I started out using standard HTTP requests to pull tick data and minute-level candlesticks. Fetching datasets for multiple symbols often took ages, plus I constantly faced API rate limits, unexpected disconnections and repeated reconnection attempts. All these issues seriously slowed down the entire backtesting workflow.

After testing different approaches, I switched to AllTick WebSocket persistent connection + dynamic subscription. This pattern cuts down connection overhead significantly and delivers much faster, more stable data fetching. In this post, I’ll walk through the problems with traditional methods, full working code, common bugs and practical optimization tips for fellow developers.

---

Table of Contents

1. Drawbacks of traditional HTTP polling
2. How dynamic WebSocket subscription works
3. Full Python implementation
4. Common issues & fixes
5. Feature limitations
6. Extra performance tweaks

---

1. Drawbacks of Traditional HTTP Polling

HTTP polling is easy to implement for beginners, but it shows clear weaknesses in batch backtesting scenarios:

• High connection overhead & long latency
  HTTP is a short-lived protocol. Every new request creates and tears down a connection. When you sequentially pull historical data for dozens of US stock symbols, accumulated connection costs lead to very slow execution.

• Unstable connections when changing subscriptions
  A common bad practice is closing the active connection and re-subscribing whenever you add or remove tracked stocks. This easily causes reconnection storms, and creates mismatches between your local subscription list and server-side status.

• Redundant data reduces overall performance
  Most market APIs return a large set of fields by default, while backtesting only requires core metrics like OHLC and volume. Without local caching, repeated API calls will further slow down data parsing and backtest execution.

The reliable solution here is to adopt WebSocket persistent connection paired with dynamic subscription management.

---

2. Dynamic WebSocket Subscription: Core Concepts & Usage

2.1 What is dynamic subscription?

Dynamic subscription means keeping one single long-lived WebSocket connection active all the time. You can add or remove stock symbols by sending dedicated commands, without disconnecting or rebuilding the network link.

Compared with HTTP polling and full re-subscription after disconnection, this design eliminates extra connection overhead entirely.

Following AllTick official documentation, US stocks use a dedicated WebSocket endpoint. All subscription operations are managed uniformly via the command cmd_id=22004.

2.2 Common Use Cases & Configuration Rules

Below I cover the most frequent scenarios you will encounter during development, along with required parameters and validation rules.

Initial bulk subscription for multiple stocks

If you subscribe many symbols separately, you will create redundant connections and waste system resources.
Use cmd_id=22004 with the subscribe action, and format symbols as Exchange:StockCode.
Validation: All targets are subscribed through only one connection, no extra links are created.

Add new symbols incrementally

Reconnecting every time you add a new stock will introduce noticeable latency.
Keep the original WebSocket alive, use cmd_id=22004 + subscribe, and append new symbol codes to the command list.
Validation: The existing connection stays active, only incremental commands are sent.

Unsubscribe selected symbols

Closing partial subscriptions often leads to status desync between local records and remote server.
Use cmd_id=22004 + unsubscribe and specify the symbols you want to remove. Remember to update your local subscription list.
Validation: The program stops receiving data from unsubscribed symbols.

Duplicate subscription requests

Sending the same symbol multiple times will bring duplicate data and increase processing load.
Even if you pass repeated codes with cmd_id=22004 and subscribe, add local deduplication logic first.
Validation: The server will not push duplicate market data.

Requests with empty symbol list

An empty code list will trigger invalid API commands and runtime errors.
Add a pre-check on your local side to block empty requests. Never send subscribe or unsubscribe commands with blank code values.

---

3. Full Working Python Code

This complete code includes connection initialization, data parsing, error handling and dynamic subscription logic. Replace YOUR_TOKEN with your real API token to get started immediately.

import websocket
import json

# Endpoint & subscription rule: AllTick official API Docs (WebSocket address specification)
# Exclusive WebSocket URL for US stocks
WS_STOCK_URL = "wss://quote.alltick.co/quote-stock-b-ws-api?token=YOUR_TOKEN"
# Local set to manage subscribed symbols for deduplication and status sync
subscriptions = set()

def on_open(ws):
    """Triggered when connection is established, run initial subscription"""
    print("WebSocket connected, starting initial subscription")
    # Sample symbols: NASDAQ AAPL and TSLA
    init_codes = ["NASDAQ:AAPL", "NASDAQ:TSLA"]
    subscriptions.update(init_codes)
    # Build standard subscription command
    sub_msg = {
        "cmd_id": 22004,
        "action": "subscribe",
        "code": init_codes
    }
    ws.send(json.dumps(sub_msg))

def on_message(ws, message):
    """Receive market data and filter invalid content"""
    if not message:
        return
    try:
        data = json.loads(message)
        code = data.get("code", "")
        price = data.get("price", 0)
        open_24h = data.get("open_24h", 0)
        # Filter empty values and abnormal data
        if not code or price <= 0 or open_24h <= 0:
            return
        print(f"Symbol: {code} | Price: {price} | 24h Open: {open_24h}")
    except json.JSONDecodeError:
        return

def on_error(ws, error):
    """Catch and print connection exceptions"""
    print(f"Connection error: {str(error)}")

def on_close(ws, close_code, close_msg):
    """Clear local records when connection closes"""
    print(f"Connection closed | Code: {close_code} | Message: {close_msg}")
    subscriptions.clear()

def add_subscribe(ws, code_list):
    """Incrementally add new symbols while reusing current connection"""
    if not code_list:
        return
    # Skip already subscribed symbols
    new_codes = [c for c in code_list if c not in subscriptions]
    if not new_codes:
        return
    subscriptions.update(new_codes)
    msg = {
        "cmd_id": 22004,
        "action": "subscribe",
        "code": new_codes
    }
    ws.send(json.dumps(msg))
    print(f"Incremental subscription completed: {new_codes}")

def remove_subscribe(ws, code_list):
    """Cancel subscription for specified symbols"""
    if not code_list:
        return
    remove_codes = [c for c in code_list if c in subscriptions]
    if not remove_codes:
        return
    for c in remove_codes:
        subscriptions.discard(c)
    msg = {
        "cmd_id": 22004,
        "action": "unsubscribe",
        "code": remove_codes
    }
    ws.send(json.dumps(msg))
    print(f"Unsubscription completed: {remove_codes}")

if __name__ == "__main__":
    # Initialize WebSocket client
    ws_app = websocket.WebSocketApp(
        WS_STOCK_URL,
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close
    )
    # Enable heartbeat every 10s to maintain stable long connection
    ws_app.run_forever(ping_interval=10)

---

4. Common Issues & Practical Fixes

Here are four frequent problems I encountered during development and deployment, with actionable solutions:

1. Callback congestion caused by high-frequency tick data
   Symptom: Massive incoming data slows down the whole program.
   Fix: Decouple data receiving and business logic. Use queues for asynchronous processing. Avoid heavy computation inside message callbacks.

2. Silent disconnection due to network fluctuation
   Symptom: No new data arrives for a long time, while the program keeps running without errors.
   Fix: Keep the heartbeat enabled. Add custom data receiving timeout logic and trigger auto-reconnection when timed out.

3. Status mismatch after frequent subscription changes
   Symptom: Still receiving data from unsubscribed symbols, or no data for newly added ones.
   Fix: Add execution locks for subscribe/unsubscribe actions to avoid concurrent commands. Verify the local subscription list after each operation.

4. Silent subscription failure from wrong symbol format
   Symptom: Program runs normally but receives no market data at all.
   Fix: Always follow the Exchange:StockCode format. Add format and spelling validation before sending subscription requests.

---

5. Feature Limitations

• ✅ Supported: Dynamically add or remove stock symbols within a single WebSocket connection.
• ❌ Not supported:
  • Sync subscription status across multiple connections
  • Fetch historical tick data using this set of commands
  • Call private commands other than cmd_id=22004

---

6. Extra Optimization Tips

WebSocket long connection already brings huge performance gains. You can optimize further with these small tweaks:

• Fetch only required fields to reduce data transfer size.
• Cache frequently used historical data locally to avoid duplicate API requests.
• Store market data in HDF5 or Parquet format for faster read & write performance.

---

Wrap Up

Switching from HTTP polling to WebSocket dynamic subscription is a straightforward but effective upgrade for US stock backtesting pipelines. It reduces connection overhead, avoids rate limits and disconnection issues, and makes your quantitative development workflow much smoother.

This implementation is lightweight, easy to deploy and fully compliant with official API specifications. Feel free to adapt it to your own trading projects.