How to Resolve Duplicate Trade Records From HK Real‑Time Stock API

💡 This post shares real‑world lessons learned while integrating Hong Kong stock market streaming APIs.

If you’ve built market‑data or quantitative tools for Hong Kong stocks, you’ve likely hit a sneaky bug:
Your locally calculated volume and candlestick data never fully matches official exchange outputs.

I ran into exactly this problem on a side project. I spent ages checking my calculation logic, convinced I had a math error. It turned out the bug was not in business logic at all. Duplicated trade messages arriving over WebSocket streams were getting consumed multiple times, corrupting all downstream metrics.

Lots of developers assume data received from an API can be used immediately. But real‑time streaming flows through many layers: network transport, message gateways, local consumers and in‑memory caches. Without idempotency safeguards, duplicate records slip into your pipeline. This issue becomes very obvious under Hong Kong market’s high‑frequency trade matching.

Where Do Duplicate Trade Messages Come From?

Three typical root causes for duplicated trades:

1. Message re‑send after network flapping
   When temporary network interruption recovers, server re‑emits messages from the disconnected period. Same trade arrives twice.

2. State lost after service restart
   If you don’t persist markers for already‑processed messages, historical trades will get re‑processed once your app restarts.

3. Lack of unique identifiers across components
   When data passes through multiple internal modules, missing unique keys lead to repeated message consumption.

⚠️ Common mistake alert: Don’t deduplicate purely using timestamps.
Multiple independent trades can happen within the same second on HK markets. Filtering only by timestamp may drop valid trades and create data gaps.

Design Deduplication Keys: Native ID or Custom Data Fingerprint

Best practice: Use the native unique trade ID provided by the API.

If the API does not expose built‑in trade IDs, combine core business fields and generate an MD5 hash as a unique fingerprint for each trade.

Fields selected: stock symbol, trade timestamp, price, volume.

import hashlib

def generate_key(trade):
    text = (
        trade["symbol"]
        + str(trade["timestamp"])
        + str(trade["price"])
        + str(trade["volume"])
    )
    return hashlib.md5(text.encode()).hexdigest()

data = {
    "symbol": "00700",
    "timestamp": "2026-08-17 10:30:20",
    "price": "380.50",
    "volume": "300"
}

print(generate_key(data))

Note: Use enough fields for fingerprint generation. Too few fields increase hash‑collision risk which misidentifies different trades as duplicates.

Integrate Deduplication Into WebSocket Stream

From an architecture perspective, decouple raw data ingestion and business computation.
Let the WebSocket client only receive streaming payloads. Deduplication runs as an independent module. Only verified, unprocessed records go downstream for candlestick generation and metric calculation.

Below is a demo snippet using AllTick API WebSocket subscription for reference:

import websocket
import json

cache_ids = set()

def on_message(ws, message):
    data = json.loads(message)
    trade_id = data.get("id")
    if trade_id in cache_ids:
        return
    cache_ids.add(trade_id)
    print(
        data.get("symbol"),
        data.get("price"),
        data.get("volume")
    )

ws = websocket.WebSocketApp(
    "wss://apis.alltick.co/ws/stock",
    on_message=on_message
)
ws.run_forever()

🚨 Important note: The in‑memory Set implementation above is for demo / local debugging only.
Do not deploy it in production. High throughput tick data will cause memory leaks and unbounded resource usage.
For production workloads, use a distributed cache with TTL to auto‑expire old fingerprints and control storage overhead.

Two Easy‑to‑Miss Production Pitfalls

Deduplication may work perfectly on your local machine but break in production, mostly due to these two issues:

1. Tune cache TTL according to market scenarios
   If TTL is too short, it cannot cover message re‑transmission windows after network reconnection, duplicates still occur.
   If TTL is too long, large amounts of useless fingerprints pile up and waste storage.
   Configure expiry time referencing HK stock trading hours and estimate maximum possible backlog after reconnect.

2. In‑memory state disappears on process restart
   Storing deduplication fingerprints in application memory means all tracking data vanishes after restart, leading to re‑processing of historical trades.
   In production, persist fingerprints inside external cache middleware to separate deduplication state from business processes.

Wrap‑up

Building financial streaming applications is more than pulling data successfully. Message idempotency is a critical underlying capability.

Deduplication code looks simple, yet it directly affects the reliability of candlestick outputs, volume statistics and backtesting results.
Hong Kong stock market has intensive intra‑day matching. Missing validation and deduplication in early stages will bring hard‑to‑reproduce data anomalies in production.

While building quantitative projects for Hong Kong equities, you can leverage Tick‑level real‑time streams from AllTick API to deepen your understanding of streaming idempotency and message deduplication.