How I Detect Missing Data Intervals in My Hong Kong Stock API Feed (And Why You Should Too)

#beginners

I write a lot of content about Hong Kong equities, and I used to assume that if my WebSocket connection was alive, my data was complete. Then I published a chart that didn’t match the official exchange feed, and a reader called me out on it. That was a wake-up call.

The problem wasn’t a broken connection. It was silent gaps — small stretches of missing ticks that happened without any error or warning. For a content creator who relies on real-time Hong Kong stock data, these gaps can quietly corrupt your analysis. In this post, I’ll show you the simple methods I now use to detect and fix them.

The Root Cause: Why Real-Time Feeds Have Gaps

Real-time market data typically arrives over a long-lived connection, with each tick carrying a timestamp. Under ideal conditions, you can reconstruct minute bars, volume profiles, and intraday patterns perfectly. But networks are never ideal. Latency spikes, brief packet loss, or even your own machine’s processing lag can cause a sequence of ticks to disappear without breaking the connection.

Here’s a real example from one of my tracking sessions:

Time	Data Status
10:00:01	Received normally
10:00:02	Received normally
10:00:03-10:00:15	No data
10:00:16	Resumed receiving

Twelve seconds of silence. In a liquid market, that could mean dozens of missed trades. If I build a chart from this feed without checking, my volume and price movement analysis will be wrong — and my readers won’t know why.

Detecting Gaps with Timestamp Analysis

The first method I use is timestamp gap analysis. For every tick, I store the original timestamp and compare it to the previous tick. If the gap exceeds a reasonable threshold, I flag it as a potential missing interval.

Here’s the core logic:

last_time = None


def check_data(timestamp):
    global last_time

    if last_time:
        gap = timestamp - last_time

        if gap > 5:
            print("Detected data gap:", gap)

    last_time = timestamp

The threshold isn’t universal. A blue-chip stock might tick every second, while a small-cap might only tick every ten seconds. I adjust the threshold based on each stock’s typical tick frequency to avoid false positives.

Adding Sequence Numbers for Stronger Detection

Timestamps are helpful, but they can miss issues when ticks are close together. That’s why I also look at message sequence numbers when the feed provides them:

20001
20002
20003
20007

The missing numbers are immediately obvious. If the API doesn’t include sequence numbers, I add a heartbeat check: periodically inspect the latest tick time, and if it hasn’t updated for too long, log the state and resubscribe. This combination catches nearly all meaningful gaps.

Putting It All Together in a Real Workflow

I separate data ingestion from validation. One process receives the ticks, and another process checks for anomalies. I’ve been using AllTick’s WebSocket feed for Hong Kong stocks, which provides raw ticks I can validate before using them in my content. Here’s a more complete implementation:

import websocket
import json


last_timestamp = None


def on_message(ws, message):
    global last_timestamp

    data = json.loads(message)

    timestamp = int(data["timestamp"])

    if last_timestamp:
        gap = timestamp - last_timestamp

        if gap > 5000:
            print("Possible missing interval:", gap)

    last_timestamp = timestamp

    print(
        data["symbol"],
        data["price"]
    )


ws = websocket.WebSocketApp(
    "wss://api.alltick.co/stock/websocket",
    on_message=on_message
)


ws.run_forever()

After finding a gap, I save the time range and later fetch historical data to fill it. This keeps my charts and analysis based on a complete record.

Recovery: Don’t Skip This Step

Detecting gaps is only the first part. If you don’t recover the missing data, your analysis is still incomplete. When I confirm a gap, I follow these steps:

Record the exact start and end time.
Request historical tick data for that range.
Verify the recovered data is continuous with my existing records and avoid duplicates.

Duplicate records can distort volume and K-line calculations just as much as missing data. So recovery needs the same level of care.

How This Has Improved My Content

Since adding these checks to my Hong Kong stock data pipeline, my charts now match the exchange feed much more closely. I no longer worry about phantom spikes or false volume dips. My readers have noticed the improvement too — they’ve told me my analysis feels more reliable and grounded.

If you’re a creator or developer working with a Hong Kong stock API, don’t trust “connected” as a sign of data health. Add timestamp gap analysis and sequence checks to your workflow. It’s a small investment that pays off in more accurate, more trustworthy content.

DEV Community