DEV Community: EmilyL

Tick Data Integration in Practice: Building a Stable Feed Pipeline

EmilyL — Thu, 03 Sep 2026 02:49:24 +0000

When tick data first enters your system, it’s easy to underestimate what’s happening. I’ve seen this firsthand as a finance lecturer mentoring a student quant team. They had a working monitor based on minute bars, and switching to tick data seemed like a small upgrade. Instead, their pipeline buckled under the load. In this post, I’ll break down why tick data behaves differently and how to design a system that can handle its rhythm.

The Startup Scenario: From Candles to a Firehose

The student team’s tool originally polled minute-level K-lines via REST. It was simple and stable. Then they added a WebSocket feed for real-time tick data. Immediately, the console flooded with timestamps, prices, and volumes. They thought adding a cache would fix the slowdown, but latency only got worse. The root cause? They treated a continuous push stream as if it were a batch of discrete responses. Tick data is an event stream, not a result set.

Why Tick Data Breaks Naive Implementations

When tick data becomes part of your core pipeline—real-time monitoring, aggregation, state triggers, or replay—its continuous nature exposes several pain points:

Unstable push frequency: bursts of ticks can overwhelm a synchronous consumer
Ordering and gaps: network jitter can reorder or drop messages
Low per-message value: you must aggregate ticks to extract useful signals
Blocking downstream: processing each tick synchronously builds latency fast

These aren’t just theoretical concerns; they’re the difference between a system that runs fine for a day and one that collapses during market open.

A Layered Approach to Consuming Tick Data

The fix is to decouple the flow. In the student project, we introduced three layers:

Access layer: maintains the WebSocket connection, handles reconnects and heartbeats
Buffer layer: uses an in-memory queue or message broker to smooth traffic spikes
Consumption layer: asynchronously aggregates, computes, and updates state

Here’s the access layer code we ended up with, stripped down but realistic:

import websocket
import json

def on_message(ws, message):
    data = json.loads(message)
    ts = data.get("timestamp")
    price = data.get("price")
    volume = data.get("volume")

    # In a real system, this would typically go into a queue or cache
    print(f"{ts} | price={price} | vol={volume}")

def on_open(ws):
    ws.send(json.dumps({
        "action": "subscribe",
        "symbols": ["US.AAPL"],
        "type": "tick"
    }))

ws = websocket.WebSocketApp(
    "wss://stream.alltick.co/v1/market",
    on_open=on_open,
    on_message=on_message
)

ws.run_forever()

Run it, and you’ll see a continuous stream in the console—a raw visualization of time-series flow. That’s when most developers realize tick data isn’t meant to be read one message at a time; it’s a stream to be processed in aggregate.

Reducing Maintenance with Unified Data Formats

Once you move beyond a single market, data format inconsistencies become a real cost. Different venues have different field names, timestamp conventions, and volume units. Writing adapters for each one bloats your access layer and introduces bugs. In several projects, I’ve found it more efficient to use a data provider that normalizes tick data across markets—such as ALLTICK API. That way, the access and logging layers stay clean, and you spend time on strategy logic instead of format wrestling. For small teams, this kind of upfront standardization is a huge long-term win.

Tick data is simple in concept but ruthless in practice. It rewards systems built for flow and punishes those designed for requests. If you’re about to integrate tick data, start by respecting its rhythm—your architecture will thank you.

Handling Time Boundaries for Hong Kong Stock API Real-Time Data

EmilyL — Wed, 02 Sep 2026 02:17:37 +0000

When our team started working with a Hong Kong stock API for a trading platform, we thought the hard part would be WebSocket reconnection or handling high message rates. It wasn't. The real pain was time boundaries — especially around market open, close, and midnight.

In this post, I'll share the approach we developed for professional trading systems and fund development teams. It's focused on extensibility and clean separation of concerns.

The Problem: Naive Time Checks Break at the Edges

Consider this common pattern:

if current_time >= time(9, 30):
    market_open = True

Looks fine, right? But at 17:00, market_open is still True. You've now labeled after-hours data as live trading data. If your risk engine or strategy consumes this, you have a silent bug.

The issue is that Hong Kong's trading day has multiple phases, not just “open” and “closed.”

Data Model: Define Sessions Explicitly

We start by splitting the day into discrete sessions:

Session	Hong Kong Time	Programmatic Treatment
Pre-open auction	09:00–09:30	Pre-market data
Morning session	09:30–12:00	Continuous trading
Lunch break	12:00–13:00	Non-trading
Afternoon session	13:00–16:00	Continuous trading
Post-close	After 16:00	Non-trading

Then we use interval logic instead of a single threshold:

def is_market_open(t):
    morning = time(9, 30) <= t < time(12, 0)
    afternoon = time(13, 0) <= t < time(16, 0)
    return morning or afternoon

This removes ambiguity at boundaries like 16:00 and 09:30.

Timezone Normalization: Always Convert to Asia/Hong_Kong

A typical Hong Kong stock API returns Unix timestamps:

{
    "symbol": "00700.HK",
    "price": 520.5,
    "timestamp": 1788226200
}

A Unix timestamp is an absolute instant, but it doesn't carry timezone info. If your server is in UTC or Singapore, you must convert:

from datetime import datetime
from zoneinfo import ZoneInfo

timestamp = 1788226200
dt = datetime.fromtimestamp(
    timestamp,
    tz=ZoneInfo("Asia/Hong_Kong")
)
print(dt)

Only after conversion do we run session logic. This prevents server timezone changes from breaking your market state.

Pre-Open Data: A Distinct State

Between 09:00 and 09:30, quotes exist, but they are not continuous trading data. We don't mark it as market_open=True.

We use a state enum:

def get_market_status(t):
    if time(9, 0) <= t < time(9, 30):
        return "pre_open"
    if time(9, 30) <= t < time(12, 0):
        return "morning"
    if time(12, 0) <= t < time(13, 0):
        return "break"
    if time(13, 0) <= t < time(16, 0):
        return "afternoon"
    return "closed"

This allows the UI to show “Pre-open” while strategies consume only morning and afternoon. Clean decoupling.

Cross-Date Issues and Trading Calendar

At 1 AM, this check:

if current_time > time(16, 0):
    status = "closed"

will correctly return closed for session state. But if you also need to know the current trading day, comparing only time is insufficient. You need the date.

Our rule: session state uses datetime.time; trading day uses datetime.date. Never mix them in one variable.

Also, Hong Kong has public holidays. Don't assume weekdays are trading days. Maintain a separate trading calendar.

K-Line Aggregation: Strict Boundaries

When you build 1-minute candles from tick data, boundaries must be precise. 09:29:59 and 09:30:00 are different minutes.

We use left-closed, right-open intervals:

09:30:00 <= tick_time < 09:31:00

This ensures each tick falls into exactly one candle.

In practice, when integrating with ALLTICK API, we convert all timestamps to Asia/Hong_Kong before session checks and K-line aggregation, following the API's documented field structure.

Our Processing Pipeline

Here's the flow we use:

API real-time data
     ↓
Parse timestamp
     ↓
Convert to Asia/Hong_Kong
     ↓
Determine trading date
     ↓
Determine session state
     ↓
Filter/classify quotes
     ↓
K-line aggregation or strategy calculation

By isolating time logic and avoiding magic numbers like 09:30 or 16:00, we made the system easier to extend to other markets.

Key Takeaways

Separate data time from trading time.
Always convert timestamps to Asia/Hong_Kong before session logic.
Use interval checks, not point comparisons.
Treat pre-open and post-close as distinct states.
Keep session state and trading day in separate variables.

Time boundaries are foundational. Get them right, and everything downstream becomes simpler.

What time-related bugs have you encountered in market data systems? Let me know in the comments!

How I Detect Missing Data Intervals in My Hong Kong Stock API Feed (And Why You Should Too)

EmilyL — Wed, 26 Aug 2026 03:01:58 +0000

I write a lot of content about Hong Kong equities, and I used to assume that if my WebSocket connection was alive, my data was complete. Then I published a chart that didn’t match the official exchange feed, and a reader called me out on it. That was a wake-up call.

The problem wasn’t a broken connection. It was silent gaps — small stretches of missing ticks that happened without any error or warning. For a content creator who relies on real-time Hong Kong stock data, these gaps can quietly corrupt your analysis. In this post, I’ll show you the simple methods I now use to detect and fix them.

The Root Cause: Why Real-Time Feeds Have Gaps

Real-time market data typically arrives over a long-lived connection, with each tick carrying a timestamp. Under ideal conditions, you can reconstruct minute bars, volume profiles, and intraday patterns perfectly. But networks are never ideal. Latency spikes, brief packet loss, or even your own machine’s processing lag can cause a sequence of ticks to disappear without breaking the connection.

Here’s a real example from one of my tracking sessions:

Time	Data Status
10:00:01	Received normally
10:00:02	Received normally
10:00:03-10:00:15	No data
10:00:16	Resumed receiving

Twelve seconds of silence. In a liquid market, that could mean dozens of missed trades. If I build a chart from this feed without checking, my volume and price movement analysis will be wrong — and my readers won’t know why.

Detecting Gaps with Timestamp Analysis

The first method I use is timestamp gap analysis. For every tick, I store the original timestamp and compare it to the previous tick. If the gap exceeds a reasonable threshold, I flag it as a potential missing interval.

Here’s the core logic:

last_time = None


def check_data(timestamp):
    global last_time

    if last_time:
        gap = timestamp - last_time

        if gap > 5:
            print("Detected data gap:", gap)

    last_time = timestamp

The threshold isn’t universal. A blue-chip stock might tick every second, while a small-cap might only tick every ten seconds. I adjust the threshold based on each stock’s typical tick frequency to avoid false positives.

Adding Sequence Numbers for Stronger Detection

Timestamps are helpful, but they can miss issues when ticks are close together. That’s why I also look at message sequence numbers when the feed provides them:

20001
20002
20003
20007

The missing numbers are immediately obvious. If the API doesn’t include sequence numbers, I add a heartbeat check: periodically inspect the latest tick time, and if it hasn’t updated for too long, log the state and resubscribe. This combination catches nearly all meaningful gaps.

Putting It All Together in a Real Workflow

I separate data ingestion from validation. One process receives the ticks, and another process checks for anomalies. I’ve been using AllTick’s WebSocket feed for Hong Kong stocks, which provides raw ticks I can validate before using them in my content. Here’s a more complete implementation:

import websocket
import json


last_timestamp = None


def on_message(ws, message):
    global last_timestamp

    data = json.loads(message)

    timestamp = int(data["timestamp"])

    if last_timestamp:
        gap = timestamp - last_timestamp

        if gap > 5000:
            print("Possible missing interval:", gap)

    last_timestamp = timestamp

    print(
        data["symbol"],
        data["price"]
    )


ws = websocket.WebSocketApp(
    "wss://api.alltick.co/stock/websocket",
    on_message=on_message
)


ws.run_forever()

After finding a gap, I save the time range and later fetch historical data to fill it. This keeps my charts and analysis based on a complete record.

Recovery: Don’t Skip This Step

Detecting gaps is only the first part. If you don’t recover the missing data, your analysis is still incomplete. When I confirm a gap, I follow these steps:

Record the exact start and end time.
Request historical tick data for that range.
Verify the recovered data is continuous with my existing records and avoid duplicates.

Duplicate records can distort volume and K-line calculations just as much as missing data. So recovery needs the same level of care.

How This Has Improved My Content

Since adding these checks to my Hong Kong stock data pipeline, my charts now match the exchange feed much more closely. I no longer worry about phantom spikes or false volume dips. My readers have noticed the improvement too — they’ve told me my analysis feels more reliable and grounded.

If you’re a creator or developer working with a Hong Kong stock API, don’t trust “connected” as a sign of data health. Add timestamp gap analysis and sequence checks to your workflow. It’s a small investment that pays off in more accurate, more trustworthy content.

How to Detect Anomalies in Stock Data API Feeds (With Python Code)

EmilyL — Tue, 25 Aug 2026 05:49:34 +0000

When I first started building real-time market data pipelines, I focused almost entirely on connection stability and update speed. But after running a complete data flow, I realized that data accuracy matters just as much. Sometimes the feed looks fine, the process keeps running, and yet when you inspect historical records, you find small deviations in prices, timestamps, or volumes.

These anomalies aren’t caused by the market. They come from the processing layer. Real-time quotes keep flowing into your system, and without validation, a few bad records can corrupt your K-lines, indicators, and strategy results. That’s why I now put data validation directly in the quote processing pipeline.

Why You Need Validation for Real-Time Quotes

Stock market data is a continuously changing stream. Every second produces new prices and trade information. During reception and processing, several types of anomalies can appear:

Timestamp order issues: new data arrives with a time earlier than the previous record.
Price changes that clearly exceed the normal range.
Volume fields missing or formatted incorrectly.
Critical fields like symbol or trading status being empty.

If these records go straight into your database, later K-line generation can drift. For example, a wrong price inside a one-minute candle can distort the high, low, and close, which then affects every technical indicator built on top.

Check the Timestamp Order First

When processing real-time quotes, I check the timestamp field first. Normally, push messages for the same symbol should be increasing over time. If the program receives a record with an earlier time, it should be temporarily filtered out to avoid breaking the downstream ordering. A simple Python check:

def check_time(last_time, current_time):
    if current_time < last_time:
        return False
    return True

In practice, you also need to account for exchange time zones, server time, and data source time. U.S. market data is especially tricky during daylight saving time transitions.

Detect Anomalous Price Moves

Price validation is another important step. A stock price won’t change without limit in an extremely short time, so you can set a threshold based on historical volatility. For example:

def check_price(old_price, new_price):
    change = abs(new_price - old_price) / old_price

    if change > 0.1:
        return False

    return True

The threshold needs to be adjusted per stock type. Large-cap stocks and high-volatility stocks shouldn’t use the same criteria.

Validate Fields After Receiving a Push

Beyond price and time, I check whether the quote data structure is complete. For a real-time push, I don’t send the record to the calculation module immediately. Instead, I confirm the following first:

symbol exists.
price is not empty.
volume is correctly formatted.
timestamp is valid.

For example, when using AllTick API’s WebSocket to receive stock quotes, I run field checks before handing the data to the next stage.

import websocket
import json


def on_message(ws, message):
    data = json.loads(message)

    if not data.get("symbol"):
        return

    if not data.get("price"):
        return

    print(data["symbol"], data["price"])


ws = websocket.WebSocketApp(
    "wss://apis.alltick.co/websocket-api/stock-websocket-interface-api/transaction-quote-subscription",
    on_message=on_message
)

ws.run_forever()

This keeps invalid records from reaching your core calculation logic.

Log Anomalies Instead of Silently Dropping Them

Many developers prefer to filter out bad data and move on. I think it’s better to keep a record. For each blocked item, store:

Symbol.
Data timestamp.
Original price.
Reason for rejection.

Later, reviewing these logs helps you determine whether the problem comes from the data source or from your own processing logic.

My Experience After Building Several Market Data Systems

The longer I work on stock quote systems, the more I care about data quality. A stock data API provides the source of truth, but validation is what makes that source usable. For real-time quotes, quantitative analysis, or automated trading, adding basic data checks reduces a lot of hidden risk.

A stable market data system isn’t only about speed. It’s about making sure every record that flows through it is reliable.

How to Handle Real-Time and Historical Data from a Stock Data Interface in One Normalized Pipeline

EmilyL — Thu, 20 Aug 2026 03:42:55 +0000

We have been reviewing brokerage tools and market data feeds for a while, and one pattern keeps coming up. Teams build a live quote dashboard and a separate historical backtest pipeline, only to discover later that the two datasets do not agree. In this post, we will share the normalization approach we use with enterprise financial data analysts.

The Problem

A typical stock data interface returns many data types: ticks, minute bars, daily bars, and so on. Real-time feeds usually contain price, volume, and timestamp. Historical bars contain open, high, low, close, and volume. If each module keeps the raw API shape, field mismatches and timezone drift start appearing as soon as you combine live and historical data.

Our Solution: A Light Transformation Layer

We insert a small conversion step before data enters storage. Every message becomes a consistent object:

market_data = {
    "symbol": "AAPL",
    "price": 225.50,
    "volume": 200,
    "timestamp": "2026-08-14T13:30:00Z"
}

With this shape, live ticks and historical bars can be stored and queried under the same rules. You do not need a custom reader for each data type.

Standardize Time from the Start

Time zone differences are easy to miss. Exchange local time versus UTC can shift every indicator. Our rule is:

Normalize all timestamps to UTC at ingestion.
Convert to exchange local time only for display or reporting.
Keep the same rule for streaming and batch data.

This one decision saves a lot of debugging later.

Connecting Real-Time and Historical Feeds

A typical stock analysis page works like this:

Load a historical window of K-lines.
Subscribe to real-time ticks.
Update the chart continuously.

The trick is ensuring the end of the historical window aligns with the start of the live stream. We pass live data through the same normalization layer first, then into cache or storage. This keeps front-end and strategy code away from raw vendor formats.

We tried this flow with a WebSocket feed for stock ticks, and the normalization looked like this:

import websocket
import json


def on_message(ws, message):
    data = json.loads(message)

    market_data = {
        "symbol": data.get("symbol"),
        "price": data.get("price"),
        "volume": data.get("volume"),
        "timestamp": data.get("timestamp")
    }

    print(market_data)


ws = websocket.WebSocketApp(
    "wss://api.alltick.co/stock/websocket",
    on_message=on_message
)


ws.run_forever()

The focus here is not just receiving data, but enforcing a consistent standard for everything entering the system.

Details Worth Planning Early

Based on our experience, plan for these:

Do not bind application fields directly to vendor response keys. Use a mapping layer.
Standardize price precision and volume units.
Handle reconnect backfill for live streams.
Define a merge policy between real-time cache and historical storage.
Keep the transformation layer simple and single-purpose.

Summary

Real-time and historical data are not separate concerns. They are different stages of the same data lifecycle. Setting up a shared schema and time rule early makes charting, strategy analysis, and backtesting more reliable. The data interface is just the starting point. The way you govern that data determines the quality of the system.

Optimizing Historical K-Line Queries from a Gold Real-Time API with a Simple Cache

EmilyL — Wed, 19 Aug 2026 06:00:03 +0000

I recently measured the performance of my gold market analysis tool and found a problem: a single backtest run was triggering 950 historical K-line requests and spending about 3.2 seconds just on data retrieval. After I introduced a local caching layer, the same task completed in 0.6 seconds.

That's an 81% improvement, and it came from a relatively simple change in how data flows through the system. Here's a detailed breakdown of the problem, the solution, and the implementation.

The Problem: Redundant Requests for Static Data

In quantitative trading, historical K-lines are accessed constantly. Whether you're calculating moving averages, Bollinger Bands, or running a full strategy backtest, you need to read candles from days or months in the past.

In my first version, every part of the code called the gold real-time API directly when it needed historical data. With a small dataset, that worked fine. But once I ran multiple strategies in parallel or swept through many parameter combinations, the latency became obvious.

The key realization was this: the API wasn't the bottleneck. The repeated network round-trips and JSON parsing were eating up time — and all for data that never changed.

Historical K-lines have a special property: once a candle is closed, it's immutable. Yesterday's 1-minute gold K-line won't change tomorrow. So fetching it repeatedly from a remote server is wasted work.

The Solution: A Two-Step Data Access Pattern

I changed the data access logic to:

Check whether the local cache already covers the requested range.
If data is missing or incomplete, fetch only the missing portion from the API.

This simple pattern eliminated most redundant traffic.

Implementing the Cache Layer

I designed two cache types based on how often the data changes.

Real-Time Data: In-Memory Cache

Real-time quotes change constantly, so I keep them in memory and only retain the most recent tick or price. This provides fast reads without persistence overhead.

Historical Data: Persistent Storage

Historical K-lines are stable, so I persist them to a local file or database. On the next startup, the program loads the cache instead of re-downloading everything.

When I store historical candles, I include these fields:

Field	Purpose
symbol	Identifies the trading instrument
timeframe	Identifies the K-line period
start and end time	Matches the requested range
OHLC data	Used for indicators and backtesting

With this metadata, I can quickly determine whether the cache satisfies a query. If only a few hours are missing, I fetch just that slice and merge it with what's already stored.

Bridging Real-Time Data and Cache

Real-time and historical data shouldn't be treated as isolated systems. If they are, you'll end up with a gap between the latest candle and the historical series.

My approach is to route real-time ticks into the cache first, then build K-lines from those ticks based on the timeframe. When a period closes, I save the completed K-line to persistent storage.

Here's an example using the AllTick API, where I receive tick data over WebSocket and store the latest price in a memory cache:

import websocket
import json
from datetime import datetime


market_cache = {}


def on_message(ws, message):
    data = json.loads(message)

    symbol = data.get("symbol")
    price = data.get("price")
    timestamp = data.get("timestamp")

    market_cache[symbol] = {
        "price": price,
        "timestamp": timestamp,
        "update_time": datetime.now()
    }

    print(symbol, price)


ws = websocket.WebSocketApp(
    "wss://api.alltick.co/ws",
    on_message=on_message
)


ws.run_forever()

Once the latest price is in the cache, subsequent K-line calculations and chart displays can read from memory directly, avoiding additional API calls.

Pitfalls and Maintenance

A cache is not a set-and-forget component.

The gold market operates nearly 24 hours, and real-time data can go stale quickly. So I use different expiration policies for real-time and historical data. Real-time prices expire fast; historical candles can be stored long-term.

Another common mistake is updating the cache by re-downloading an entire range when only a small segment is missing. Instead, fetch just the missing part and merge it. For a long-running system, this difference accumulates significantly.

If you need multiple strategies to read the same data concurrently, consider upgrading to Redis so different processes can share one cached copy. That's the natural next step when the system scales.

Conclusion: Data Flow Matters as Much as API Speed

This optimization changed how I think about performance in market data systems. The API speed is only one factor. How data moves through your application is equally important.

Real-time data provides fresh prices; the cache eliminates redundant reads. Together, they keep backtesting and indicator calculations stable.

If your application queries historical K-lines frequently, treat caching as a core part of the data pipeline — not an optional enhancement. Design it early, and you'll have a much easier time scaling to more symbols and larger datasets later.

Stock API Minute‑to‑Daily Bar Aggregation: How We Solved Timezone Offset Issues in Our Trading Data Pipeline

EmilyL — Thu, 13 Aug 2026 02:57:28 +0000

Hey devs! Today I want to share a data engineering war story that will resonate with anyone who has ever built a financial data pipeline. Our team supports a group of cross‑border quantitative traders who analyze US equities. To keep infrastructure costs lean, we fetch minute‑level OHLC data from a stock API and aggregate it into daily bars ourselves. Everything was fine until one Monday morning when a strategist asked, “Why does the daily open for AAPL on July 1 not match the exchange?” That question kicked off a deep dive into exchange timezones, daylight saving rules, and trading calendars.

Scenario: Building a Cost‑Efficient Daily Bar Factory

We work with independent investors and small trading desks who operate across multiple markets. Paying for premium daily bar feeds for every region would eat up most of their research budget. So we architected an internal data service that consumes an affordable stock API for minute data and produces daily bars on‑the‑fly. The design is simple: ingest minute bars, group by date, compute OHLC. But that simplicity hides a critical assumption—that the date attached to each minute bar already belongs to the correct trading session.

The Core Pain: UTC Grouping Splits Trading Sessions

The stock API we use returns timestamps in UTC. For US stocks, trading happens in Eastern Time. In daylight saving, the market opens at 09:30 ET (13:30 UTC) and closes at 16:00 ET (20:00 UTC). If we group by the UTC date, the bars between 19:00 and 20:00 UTC belong to the next UTC day. The result: a single trading day gets split across two daily bars. Here’s the simple illustration that made everything click for us:

Time Type	Corresponding Time
Eastern Trading Time	2026-07-01 09:30
UTC Time	2026-07-01 13:30

Once you see this, it’s obvious. But in a large pipeline that processes hundreds of symbols, the error manifests as sporadic opening‑price jumps that are maddeningly hard to trace.

Solution: A Time‑Normalization Layer Before Any Aggregation

We refactored the pipeline to treat time conversion as a mandatory preprocessing step. No piece of code that computes OHLC ever sees a raw UTC timestamp. The flow now looks like:

Parse the original timestamp from the API response.
Load the target exchange timezone from a configuration map (for US stocks, America/New_York).
Use Python’s zoneinfo to convert to local time, fully accounting for DST.
Assign a trading date based on the converted timestamp and session rules.

The core conversion logic is self‑contained:

from datetime import datetime
from zoneinfo import ZoneInfo

utc_time = datetime.strptime(
    "2026-07-01 13:30:00",
    "%Y-%m-%d %H:%M:%S"
)

utc_time = utc_time.replace(
    tzinfo=ZoneInfo("UTC")
)

market_time = utc_time.astimezone(
    ZoneInfo("America/New_York")
)

print(market_time)

This snippet runs on every batch load, guaranteeing that all minute bars are aligned to the exchange clock before aggregation.

Extending the Solution: Trading Sessions and Calendars

Timezone conversion alone isn’t enough. Many APIs include pre‑market and after‑hours trades. If those get mixed into the daily bar, your technical indicators will quietly degrade. We therefore added a session filter:

Only minute bars between 09:30 and 16:00 Eastern are used for the standard daily bar.
Extended‑hours data is routed to a separate analytics store.
A trading calendar service dynamically adjusts for half‑days and holidays, so the system never assumes a fixed bar count.

Real‑Time Streaming Consistency

For live trading dashboards, we ingest real‑time ticks through a WebSocket. To keep real‑time bars identical to historical ones, the tick processor reuses the same time‑normalization module. We use a low‑latency, budget‑friendly feed from AllTick for our US equity streams, and the integration was seamless because we had already solved the time problem generically.

Here is our real‑time tick handler. Note how time conversion is the very first operation:

import websocket
import json
from datetime import datetime
from zoneinfo import ZoneInfo

def on_message(ws, message):
    data = json.loads(message)

    trade_time = data["tradeTime"]

    dt = datetime.strptime(
        trade_time,
        "%Y-%m-%d %H:%M:%S"
    )

    market_time = dt.replace(
        tzinfo=ZoneInfo("America/New_York")
    )

    print(
        data["symbol"],
        data["price"],
        market_time
    )

ws = websocket.WebSocketApp(
    "wss://api.alltick.co/stock/websocket",
    on_message=on_message
)

ws.run_forever()

Lessons Learned and Practical Tips

Throughout this project, we’ve built a small checklist that might save you some pain:

Normalize early. If you consume data from multiple stock APIs, agree on a single timezone standard before any merge.
Preserve raw timestamps. Store them as a debug column. When an analyst questions a daily bar, the raw value is your audit trail.
Never hard‑code bar counts. Markets have early closes. Use a calendar to determine expected bar counts dynamically.
Encapsulate time logic. A shared time‑conversion module used by batch and real‑time paths eliminates whole classes of bugs.

Conclusion

What started as a minor discrepancy in a daily bar turned into a comprehensive refinement of our entire data architecture. The fix wasn’t in the aggregation math; it was in the invisible timestamp semantics that preceded it. For any developer building financial tools on top of a stock API, my honest advice is to obsess over timezone correctness early. Once your temporal foundation is solid, the candlesticks you generate will finally reflect the true market narrative, and your traders can focus on strategy instead of data forensics.

Tutorial: Fix US Stock API Candlestick Gaps with Session-Aware Aggregation

EmilyL — Wed, 12 Aug 2026 06:02:35 +0000

Have you ever built a candlestick chart for US equities, only to find that the pre-market and after-hours segments look like a broken zipper? We’ve been there while powering financial bloggers’ data dashboards. The good news is that the issue rarely lies in your frontend library—it’s almost always a data modeling problem. In this tutorial, we’ll share how we re-architected our US stock market API pipeline to produce continuous candlestick charts across all trading sessions.

Understanding Why Extended Hours Create Visual Breaks
US stocks trade outside the 09:30–16:00 ET regular window, with official pre-market and after-hours sessions. These sessions contain real trades, but at drastically lower frequencies. A typical implementation that chops the day into fixed 5-minute windows will encounter many intervals with zero trades during extended hours. If your code simply skips those intervals, the time series compresses and the chart displays a price jump. The fix is to stop treating all ticks equally and start respecting the market’s session structure.

Step 1: Force Time Consistency in Your Data Pipeline
We learned early that not all US stock APIs speak the same time language. Some return UTC, others Eastern Time. To avoid candlestick shifts, we normalize every incoming tick to UTC at the ingestion layer. We retain the original exchange timestamp in a separate field for debugging. When rendering a chart for end users, we convert back to Eastern Time. This one practice eliminates entire categories of offset bugs.

Step 2: Label Every Tick with Its Trading Session
We enrich each tick with a session identifier based on its Eastern Time timestamp:

Trading Session	Time Range (ET)	Handling Method
Pre-Market	Before 09:30	Recorded separately
Regular Trading	09:30–16:00	Normal aggregation
After-Hours	After 16:00	Processed independently

With this tagging, your aggregation logic can branch. Need a pure regular-session view? Filter by the “Regular Trading” tag. Building a full-day continuous chart? Aggregate each session separately and then merge by timestamp. The chart’s time axis stays linear because no interval is discarded.

Step 3: Aggregate Raw Ticks Instead of Using Pre-Built Candles
In production, we avoid consuming pre-aggregated candlestick data from any API. Instead, we stream raw tick data over WebSocket. By tapping a service like AllTick API, we receive real-time US stock trades with precise timestamps, and we perform the candlestick construction in our own code.

import websocket
import json

def on_message(ws, message):
    # Parse incoming market data
    data = json.loads(message)

    symbol = data.get("symbol")      # Stock ticker
    price = data.get("price")        # Latest trade price
    timestamp = data.get("timestamp") # Trade timestamp

    print(symbol, price, timestamp)

# Establish a WebSocket connection
ws = websocket.WebSocketApp(
    "wss://api.alltick.co/stock/websocket",
    on_message=on_message
)

# Keep listening for new trades
ws.run_forever()

Inside the aggregation loop, we check the timestamp against the session boundaries, place the trade into the appropriate candlestick bucket, and emit completed candles once the bucket’s time is up. This gives us complete command over how pre-market, regular, and after-hours candles merge.

Step 4: Define the Merge Rules Upfront
Before showing charts to your audience, nail down the business rules. Will your daily bar incorporate after-hours prices? Do 5-minute charts during the pre-market appear as standalone segments or blend into the regular session? We document these choices with the content creators we support, so every chart they publish aligns with their analytical narrative.

Result: Charts Your Users Can Trust
Since implementing this pipeline, the financial writers we work with have stopped receiving “why does this chart look broken?” replies. Their content is more authoritative because the underlying data respects actual market structure. If you’re building a US stock chart application, give session-aware aggregation a try—you’ll clean up those extended-hours artifacts for good.

Multi-Asset WebSocket Market Data APIs for Stocks, Forex, Crypto & Commodities: A 2026 Technical Comparison with AllTick

EmilyL — Thu, 06 Aug 2026 06:44:51 +0000

Introduction

Engineers building real-time dashboards, algorithmic trading systems, or market-screening tools frequently hit the same pain points: fragmented APIs that force you to stitch together WebSocket feeds from disparate providers, inconsistent data schemas across asset classes, opaque rate‑limit models that cripple backtesting, and the operational burden of managing multiple authentication tokens and connection life‑cycles. In 2026, the landscape still demands a careful evaluation of latency, coverage, and developer ergonomics before committing to a market data backbone.

This article provides a technical comparison of three public APIs — AllTick, Finnhub, and Binance — with a focus on their real‑time WebSocket capabilities and REST‑based historical data retrieval. AllTick serves as the primary implementation reference because it offers native multi‑asset coverage under a single API contract, making it a representative example for workflows that span equities, forex, crypto, and commodities.

Selection Criteria

Three core benchmarks guide the evaluation:

Real‑Time Data Delivery & Latency

How quickly does a tick travel from the exchange to the subscriber? Includes WebSocket protocol efficiency, geo‑proximity of gateway clusters, and observed end‑to‑end latency.
Asset Class Coverage & Data Granularity

The breadth of instrument types (stocks, forex pairs, crypto, commodities) and the finest available resolution (true tick‑by‑tick vs. aggregated 1‑minute bars).
API Integration Effort & Developer Experience

Onboarding friction: authentication model, SDK availability, WebSocket subscription logic, rate‑limit transparency, and historical data retrieval mechanics.

Comparative Overview

Mini‑Reviews

AllTick – A unified market data API delivering low‑latency WebSocket streams and REST endpoints for equities, forex, crypto, and commodities, designed to reduce the number of vendor integrations in multi‑asset applications.
Finnhub – A developer‑friendly API with strong US equity fundamentals and a generous free tier, though forex and crypto feeds are comparatively light and heavily rate‑limited.
Binance API – The de‑facto crypto‑native data pipe, offering exhaustive tick‑level streams, deep historical order book snapshots, and virtually unrestricted public data access, but scoped exclusively to digital assets.

Comparison Matrix

Feature	AllTick	Finnhub	Binance API
Free‑Tier Rate Limits	30 REST requests/min, 1 WebSocket connection, up to 10 symbols	60 REST calls/min, 1 WebSocket connection, 50 symbols (US equities only on WS free)	Public market data: no strict request caps; up to 5 WebSocket connections, 200 streams each
Real‑Time Latency	Typically <100 ms (Asian gateway)	100‑200 ms (US equities via Finnhub WS)	<100 ms (Binance cloud; edge clusters globally)
Data Granularity	Tick, 1m, 5m, 15m, 30m, 1h, 4h, daily	1m, 5m, 15m, 30m, 1h, daily (tick only for US stocks on paid plans)	Tick (trade/aggTrade), 1m, 3m, 5m, …, daily
Supported Protocols	REST + WebSocket	REST + WebSocket	REST + WebSocket
Historical Data Depth	Up to 10 years (stocks), 5 years (forex), full exchange history (crypto); free tier includes recent 12 months	1 year for free tier; extended history on paid plans	Full exchange history (e.g., Binance Spot since 2017)
Ideal Use Cases	Multi‑asset dashboards, cross‑market arbitrage scanners, brokerage back‑offices	US stock sentiment analysis, lightweight portfolio tracking	Crypto trading bots, deep order‑book analytics, DeFi oracles

AllTick data in the matrix reflects the standard public plan; enterprise tiers relax rate limits and extend connectivity options.

Implementation Guide

The following examples use the AllTick API to demonstrate typical market data workflows. All code is production‑ready Python 3.10+ and relies only on standard libraries plus requests and websocket-client.

Authentication – Every request must include the API key in the header X-API-Key. Free keys are obtainable from the AllTick developer portal.

1. REST API – Fetch Candlestick (K‑Line) Data

The /klines endpoint returns OHLCV bars for a given instrument and interval. Parameters:

code – Instrument identifier (e.g., "AAPL.US", "EUR/USD", "BTC/USDT", "XAU/USD").
kline_type – Resolution: "1m", "5m", "15m", "30m", "1h", "4h", "1d", etc.
count – Number of bars to return (max 1000 per call).

import requests
from datetime import datetime, timezone

API_KEY = "YOUR_ALLTICK_API_KEY"
BASE_URL = "https://api.alltick.io/v1"

def fetch_klines(code: str, kline_type: str = "1d", count: int = 10):
    url = f"{BASE_URL}/klines"
    headers = {"X-API-Key": API_KEY}
    params = {
        "code": code,
        "kline_type": kline_type,
        "count": count
    }

    resp = requests.get(url, headers=headers, params=params)
    resp.raise_for_status()
    data = resp.json()

    if data.get("code") != 0:
        raise RuntimeError(f"API error: {data.get('msg')}")

    for bar in data["data"]:
        ts = datetime.fromtimestamp(bar["t"] / 1000, tz=timezone.utc)
        print(f"{ts} O:{bar['o']} H:{bar['h']} L:{bar['l']} C:{bar['c']} V:{bar['v']}")

# Example: last 10 daily bars for Apple Inc.
fetch_klines("AAPL.US", "1d", 10)

Workflow note – The response envelope always contains "code":0 on success, an array of OHLCV objects under "data", and an optional "total" field when a time range is queried (see historical retrieval). The timestamp t is epoch milliseconds in UTC.

2. WebSocket – Real‑Time Tick Data

AllTick’s WebSocket gateway supports concurrent subscription to multiple instruments across asset classes. A single connection can carry equity quotes, forex prices, crypto trades, and commodity ticks.

Connection & authentication – Pass the API key as a query parameter. The gateway returns a heartbeat every 30 seconds; clients should implement a reconnection back‑off.

import json
import time
import websocket

WS_URL = "wss://ws.alltick.io/stream"

def on_message(ws, message):
    tick = json.loads(message)
    # Filter out heartbeats
    if tick.get("type") == "tick":
        print(f"{tick['code']} @ {tick['time']}  price={tick['price']}  vol={tick['volume']}")

def on_error(ws, error):
    print(f"WebSocket error: {error}")

def on_close(ws, close_status_code, close_msg):
    print("Connection closed – reconnecting in 5s...")
    time.sleep(5)
    start_stream()

def on_open(ws):
    # Subscribe to multiple instruments
    subscribe_msg = {
        "action": "subscribe",
        "symbols": ["AAPL.US", "EUR/USD", "BTC/USDT", "XAU/USD"]
    }
    ws.send(json.dumps(subscribe_msg))
    print("Subscribed to real-time ticks")

def start_stream():
    ws = websocket.WebSocketApp(
        f"{WS_URL}?token={API_KEY}",
        on_open=on_open,
        on_message=on_message,
        on_error=on_error,
        on_close=on_close
    )
    # Run forever with automatic ping/pong (websocket-client handles ping)
    ws.run_forever(ping_interval=30, ping_timeout=10)

start_stream()

Tick object structure – Each tick message includes code, price, volume, time (epoch ms), and an optional bid/ask spread for forex/commodities. The gateway guarantees ordered delivery within a symbol.

Architecture decision – Opening a single WebSocket with multi‑symbol subscription reduces the number of file descriptors and simplifies application‑level reconnection logic compared to one connection per symbol (the pattern required by many legacy APIs). AllTick enforces a maximum of 10 symbols on the free tier; paid plans lift this limit.

3. Historical Data Retrieval

For backtesting or down‑sampling, you often need large chunks of archived data. The /history/kline REST endpoint accepts a time window and returns paginated results.

Parameters:

code, kline_type – same as before.
start_time / end_time – epoch milliseconds in UTC.
limit – batch size (max 1000). If the total number of bars in the window exceeds limit, the response includes a total field and you must paginate using the last returned timestamp as the new start_time.

def fetch_historical_klines(code: str, kline_type: str, start_ms: int, end_ms: int):
    url = f"{BASE_URL}/history/kline"
    headers = {"X-API-Key": API_KEY}
    all_bars = []

    while start_ms < end_ms:
        params = {
            "code": code,
            "kline_type": kline_type,
            "start_time": start_ms,
            "end_time": end_ms,
            "limit": 1000
        }
        resp = requests.get(url, headers=headers, params=params)
        resp.raise_for_status()
        data = resp.json()
        if data["code"] != 0:
            raise RuntimeError(data["msg"])

        batch = data["data"]
        if not batch:
            break

        all_bars.extend(batch)
        # Set next start to timestamp of last received bar + 1 ms to avoid duplicates
        start_ms = batch[-1]["t"] + 1

        # Respect rate limits: free tier allows 30 req/min
        time.sleep(2.1)

    return all_bars

# Example: 1-minute bars for EUR/USD from August 1 to August 5, 2026
import datetime
start = int(datetime.datetime(2026, 8, 1, tzinfo=timezone.utc).timestamp() * 1000)
end   = int(datetime.datetime(2026, 8, 5, tzinfo=timezone.utc).timestamp() * 1000)

bars = fetch_historical_klines("EUR/USD", "1m", start, end)
print(f"Retrieved {len(bars)} 1m bars")

Error handling – The function pauses 2.1 seconds between pages to stay within the 30‑request‑per‑minute window. For production, parse the X-RateLimit-Remaining header when available and implement an adaptive wait.

Workflow advantage – The same endpoint serves stocks, forex, crypto, and commodities, and the response schema remains identical. This uniformity lets you reuse pagination logic across all asset types with zero code changes.

API Docs：https://apis.alltick.co/

GitHub：https://github.com/alltick/alltick-realtime-forex-crypto-stock-tick-finance-websocket-api

Reconstructing a Cryptocurrency Order Book from Real-Time API Incremental Updates — A Practical Guide

EmilyL — Thu, 06 Aug 2026 03:02:33 +0000

When we started building out our crypto trading infrastructure, we quickly ran into a challenge that sounds simple but is actually a deep engineering problem: turning a firehose of incremental order book updates from a cryptocurrency API into a continuously accurate local representation of the market. This post is the guide we wish we’d had back then, written from the perspective of a team of independent high-frequency traders. We’ll cover the requirements, the data pain points, the specific product-level solutions we built, and how it all fits into real industry applications.

What Problem Are We Solving?

We need a local order book that accurately reflects the exchange’s live state. Every strategy — from quoting to statistical arbitrage to liquidity analysis — depends on this ground truth. But the API doesn’t send us a full picture; it sends small incremental patches.

Data Pain Points: Understanding What You’re Actually Receiving

Incremental Updates Are Instructions, Not States

A full order book contains every active bid and ask level with their respective quantities. However, pushing the entire book on each change would be prohibitively bandwidth-heavy. Instead, cryptocurrency APIs use incremental feeds that deliver only the modified levels. You’ll see something like:

Direction	Price	Quantity Change
Buy	65000	+0.5
Sell	65010	-1

Think of this as a log entry: “Apply this delta.” You cannot interpret it in isolation; you must maintain your own state and apply each delta in sequence. If you treat each delta as an independent event, the order of application can get scrambled, and your local book will drift from reality.

Sequence Numbers Are Your Only Safeguard

Network transport means messages can arrive out of order. A later message might reach your server before an earlier one. If you apply them by arrival time, the state becomes logically corrupted. Our solution is strict sequence-number discipline. The API provides a sequence or updateId with every message. Our rule:

Get a full snapshot and record its sequence number.
Buffer all subsequent deltas.
Only apply deltas whose sequence number is exactly one greater than our current state.
If a sequence gap appears, discard the local state and re-fetch a fresh snapshot.

This ensures we never build on a broken foundation.

Product Functionality: Designing the Local Book and Connection

Use a Dictionary, Not a List

Early on, we used arrays for price levels. As the number of levels grew, update performance tanked. Now we use dictionaries (maps) keyed by price:

order_book = {
    "bids": {
        65000: 1.5,
        64999: 2.0
    },
    "asks": {
        65001: 1.8,
        65002: 3.1
    }
}

On each delta: if new quantity > 0, set the price key; if quantity == 0, delete the key. This structure makes getting best bid/ask and calculating depth extremely fast.

WebSocket for Low-Latency Feeds

HTTP polling introduces unacceptable latency for order book updates. We always use WebSocket connections. While integrating one feed, we based our initial handler on the AllTick API WebSocket market data pattern. The core processing loop is:

import websocket
import json

order_book = {
    "bids": {},
    "asks": {}
}

def update_order_book(data):
    for item in data.get("bids", []):
        price = float(item["price"])
        volume = float(item["volume"])
        if volume == 0:
            order_book["bids"].pop(price, None)
        else:
            order_book["bids"][price] = volume
    for item in data.get("asks", []):
        price = float(item["price"])
        volume = float(item["volume"])
        if volume == 0:
            order_book["asks"].pop(price, None)
        else:
            order_book["asks"][price] = volume

def on_message(ws, message):
    data = json.loads(message)
    if data.get("symbol") == "BTCUSDT":
        update_order_book(data)
        print(order_book)

ws = websocket.WebSocketApp(
    "wss://apis.alltick.co/websocket-api",
    on_message=on_message
)
ws.run_forever()

This is the foundation. Wrap it with sequence validation, heartbeat checks, and automatic snapshot recovery on disconnect for a production-grade system.

Industry Applications: Where Accurate Depth Matters

We use this book for market making, arbitrage signals, and real-time liquidity monitoring. A small consistent error in depth data can gradually erode profitability. Two additional notes from our production experience:

After a WebSocket disconnect, the local book is outdated. Reconnect logic must fetch a fresh snapshot first, then resume deltas.
Price precision: avoid floating-point keys. Convert all prices to integers based on the tick size to prevent matching errors.

Accurate order book reconstruction is not just about receiving data; it’s about synchronizing a distributed state. Get this right, and your trading strategies have a solid foundation.

Building a Reliable XAUUSD Tick Pipeline with a Precious Metals API: Our Battle with Daylight Saving Time

EmilyL — Wed, 05 Aug 2026 03:31:16 +0000

What happens when your gold trading strategy suddenly breaks in summer

It was one of those late-night debugging sessions that every cross-border fintech startup knows too well. Our team had just rolled out an internal platform that combined a precious metals API with a custom backtesting engine for XAUUSD tick data. The system performed beautifully during the winter months. Our momentum-based intraday strategies were hitting win rates and Sharpe ratios that made our investors raise their eyebrows — in a good way.

Then, without any code changes, the models began to deteriorate. Signals that should have triggered right at the New York open were instead firing one hour late. Live simulations diverged from backtests in ways we couldn’t explain. As the founders, we were staring at a crisis of data integrity — and we needed to fix it before our track record suffered.

Unpacking the problem: when time becomes a variable

We traced every failure back to one assumption we’d made early on: that converting timestamps from the precious metals API to UTC was as simple as adding a constant offset. The API returns data in US Eastern Time. We’d hard-coded Eastern Time as “UTC minus 5 hours.” That’s true in winter. It’s false during daylight saving time.

The error is subtle but devastating at tick resolution. Here’s a concrete example:

Time Status	US Eastern	UTC
Standard Time	09:30	14:30
Daylight Saving	09:30	13:30

A tick generated during the summer at 13:31 UTC was being wrongly classified under the 14:31 UTC minute candle, which completely reshuffled the order book snapshots our strategy relied on. Our precious metals API was feeding us perfect prices — our own code was misplacing them in time.

Engineering the fix: a UTC-centric data pipeline

We rebuilt our tick ingestion pipeline around a single principle: UTC is the only timezone that exists inside the system.

Regardless of the source format, every timestamp is transformed into UTC at the earliest possible moment — before it enters the database, before it reaches the K-line aggregator, and certainly before any strategy code sees it. Python’s zoneinfo module became our trusted ally because it dynamically resolves DST transitions based on the operating system’s timezone database.

The conversion function we now use everywhere:

from datetime import datetime
from zoneinfo import ZoneInfo

time_str = "2026-06-05 09:30:00"

new_york = ZoneInfo("America/New_York")
utc = ZoneInfo("UTC")

dt = datetime.strptime(
    time_str,
    "%Y-%m-%d %H:%M:%S"
)

local_time = dt.replace(tzinfo=new_york)
utc_time = local_time.astimezone(utc)

print("UTC time:", utc_time)

For real-time data, we applied the identical logic in our WebSocket message handler. When we connected to a reliable market data provider such as AllTick for live XAUUSD ticks, the first thing on_message does is convert the timezone:

import websocket
import json
from datetime import datetime
from zoneinfo import ZoneInfo

def on_message(ws, message):
    data = json.loads(message)

    price = data.get("price")
    trade_time = data.get("tradeTime")

    eastern = ZoneInfo("America/New_York")
    utc = ZoneInfo("UTC")

    dt = datetime.strptime(
        trade_time,
        "%Y-%m-%d %H:%M:%S"
    )

    utc_time = dt.replace(
        tzinfo=eastern
    ).astimezone(utc)

    print(price, utc_time)


ws = websocket.WebSocketApp(
    "wss://api.alltick.co/ws",
    on_message=on_message
)

ws.run_forever()

From that point forward, K-line aggregation, strategy logic, and storage all operate on UTC. The rendering layer handles conversion to any display timezone.

The business impact: reclaimed time and regained trust

Before this fix, each DST transition season required two engineers to spend several days manually reviewing timestamps, correcting historical datasets, and re-running backtests. That’s a significant annual drain on a lean startup’s resources. Now, those days are fully reclaimed for feature development and alpha research.

But the larger benefit is confidence. When your precious metals API delivers tick data and your strategies consume it, you must be absolutely certain that every data point sits in the correct temporal context. We learned the hard way that a one-hour offset can be more damaging than a pricing error. Fixing it gave us a foundation we can scale on — no matter how many strategies or markets we add.

Handling Out‑of‑Order Level2 Messages from a US Stock API – A Practical Guide

EmilyL — Thu, 30 Jul 2026 02:52:24 +0000

As developers supporting quantitative trading desks, we’ve spent countless hours debugging order book inconsistencies. The root cause almost always traces back to one issue: message sequencing. In this post, we’ll share our battle‑tested approach to processing Level2 data from a US Stock API, focusing on how we maintain a reliable order book despite network‑induced reordering.

Client Requirements: Zero Tolerance for State Errors

Our primary users are professional traders and fund quant developers. They consume our Level2 feed to build real‑time liquidity models and execute automated strategies. Their non‑negotiable demand: the local order book must be a bit‑perfect replica of the exchange’s limit order book at any given moment. A single mis‑applied delta – for example, a modification applied before its corresponding add – can skew the entire price ladder and trigger incorrect trades. We learned this when a client reported a persistent spread miscalculation; after investigation, we found that out‑of‑order cancellation messages were the culprit.

The Pain Point: Network Jitter and Packet Reordering

Unlike consolidated tape data, Level2 streams consist of discrete order‑state transitions – insert, update, delete. These operations are inherently dependent. When you fetch Level2 via a typical US Stock API over WebSocket, you’re at the mercy of internet routing. It’s entirely possible for a later event to arrive before an earlier one. For instance, the exchange emits:
Insert (seq=100) → Update (seq=101) → Delete (seq=102)
But our receiver may see:
Update (101) → Insert (100) → Delete (102)
Applying them in arrival order would cause an “order not found” error on the update, and later an invalid insert. This becomes especially frequent during high‑volatility periods when message rates exceed 10,000 per second.

Sequence Numbers as the Single Source of Truth

We quickly dismissed timestamp‑based sorting – it’s unreliable due to clock skew and low resolution. Instead, we rely on the monotonic sequence field included in each Level2 message. Here’s an example payload:

{
 "symbol":"AAPL",
 "price":185.2,
 "volume":300,
 "sequence":10001
}

We maintain a last_processed_seq variable. If the incoming sequence equals last_processed_seq + 1, we apply the delta. If it’s greater, we declare a gap and trigger a full snapshot recovery. If it’s smaller, we discard it as duplicate. This logic is simple yet effective – it catches missing messages immediately, preventing silent data corruption.

Architecture: Snapshot + Incremental with Recovery

Our production system follows a “snapshot + incremental” model:

Initialization: Fetch a complete order book snapshot (all price levels and aggregated sizes) via a REST endpoint.
Streaming: Open a WebSocket connection to receive incremental updates.
Validation: For each delta, check sequence continuity.
Recovery: On any gap, pause incremental processing, fetch a fresh snapshot, replace the local book, and reset the sequence counter.

We also implement a small delay buffer: when we receive a message with a sequence number slightly ahead (e.g., we expect 10003 but get 10004), we hold it for up to 50ms to see if 10003 arrives. If it does, we reorder and apply. If not, we fetch a snapshot. This reduces unnecessary full refreshes.

Code Walkthrough – Python WebSocket Listener

Below is the core implementation we use as a starting point. The example uses a common WebSocket endpoint (we’ve integrated with various providers; the pattern is identical). Notice the sequence validation in the callback:

import websocket
import json

last_sequence = 0

def on_message(ws, message):
    global last_sequence

    data = json.loads(message)

    seq = data.get("sequence")

    if seq and last_sequence:
        if seq != last_sequence + 1:
            print("Data gap detected – re‑sync required")

    last_sequence = seq

    print(
        data.get("symbol"),
        data.get("price")
    )

ws = websocket.WebSocketApp(
    "wss://api.alltick.co/ws",
    on_message=on_message
)

ws.run_forever()

For production, we add:

Automatic reconnection with exponential backoff.
A thread‑safe cache for out‑of‑order messages.
Health checks that compare our local book against occasional snapshot hashes.
Metrics to monitor gap frequency and recovery latency.

Lessons Learned

The hardest part of Level2 processing isn’t writing the update logic – it’s ensuring the update order is correct. We’ve come to treat sequence validation as our primary defense against data corruption. If you’re building any system that consumes depth‑of‑market data, invest in this foundation first. Accurate prices follow accurate sequences – never the other way around.