DEV Community: EmilyL

Reconstructing Historical Order Books with a Crypto API – A Step‑by‑Step Guide

EmilyL — Thu, 23 Jul 2026 03:44:56 +0000

We’ve all been there: you’re debugging a backtest, and the strategy made a terrible decision at 14:32:07. You check the candlestick—nothing unusual. You check the volume—normal. So what caused the slip? The answer almost always lies in the order book. But how do you get the order book as it was at that exact millisecond, long after the moment has passed?

In this tutorial, we’ll show you how to use a crypto API’s WebSocket stream to capture and store order‑book updates, so you can later retrieve a full snapshot for any given timestamp. We’ll share our own architecture, code snippets, and the lessons we’ve learned along the way.

Defining the Order‑Book Snapshot

A snapshot is a point‑in‑time representation of all active limit orders. It consists of:

Bids – buy orders with prices and quantities.
Asks – sell orders with prices and quantities.
Timestamp – when the snapshot was taken.
Symbol – the market pair.

Example (BTCUSDT):

{
  "symbol": "BTCUSDT",
  "timestamp": 1784188200000,
  "bids": [
    ["65000", "2.5"],
    ["64990", "1.8"]
  ],
  "asks": [
    ["65010", "1.2"],
    ["65020", "3.1"]
  ]
}

With this data, we can analyse depth changes preceding price moves—essential for any serious quant.

The Problem: No Direct Historical Query

We scoured the documentation of every major crypto API. None provides a “get historical order book” method. The reason is obvious: order books mutate every few milliseconds, and storing all versions is prohibitively expensive. So we must design a system that subscribes to real‑time updates and persists them.

Comparing Data Collection Methods

Approach	Accuracy	Complexity	Best For
HTTP REST polls	Low (misses intra‑interval changes)	Low	Casual monitoring
WebSocket stream	High (captures every event)	Medium	Backtesting, research

We chose WebSocket because our backtests require event‑level precision. Let’s see how to set it up.

Connecting to a Stream (Using AllTick API as an Example)

We used a provider that offers a straightforward WebSocket endpoint—we’ll call it AllTick API. The connection code is minimal:

import websocket
import json

url = "wss://apis.alltick.co/websocket-api/stock-websocket-interface-api/transaction-quote-subscription"

def on_message(ws, message):
    data = json.loads(message)
    print(data)

ws = websocket.WebSocketApp(
    url,
    on_message=on_message
)

ws.run_forever()

In reality, we don’t just print. We parse the message, identify whether it’s a full snapshot or an incremental update, and then feed it into our storage pipeline. We also maintain a local in‑memory copy of the order book.

Storage Design: Choose Your Granularity

The volume of order‑book data is large. We recommend storing different densities for different purposes:

Purpose	Storage Recommendation
Simple trend review	Full snapshot every 5 seconds
Strategy backtesting	Full snapshot every 200 ms + delta logs
High‑frequency research	All deltas, no snapshots (reconstruct later)

We personally use a hybrid: we save full snapshots every second and archive all deltas. To reconstruct a specific time, we load the nearest snapshot and replay the deltas. Always use the exchange’s timestamp—not your local time—as the authoritative ordering key.

Critical Operational Checks

Over time, we’ve compiled this checklist to keep our data clean:

Timestamp gap detection – if the interval between two messages exceeds a threshold (say 100 ms), we assume packet loss and flag that range.
Reconnection logic – after a WebSocket disconnect, we first fetch a full snapshot (via REST) to re‑establish the baseline, then resume the delta stream.
Message type handling – we strictly separate snapshot (overwrite) from update (merge) to avoid double‑counting.
Sanity filters – we reject prices ≤0, quantities ≤0, and any value that deviates wildly from the current mid‑price.

These checks have drastically reduced our debugging time.

Why We Keep Building This Archive

We believe that the true power of a crypto API lies not in real‑time ticks, but in the historical depth it enables us to build. When we encounter a black‑swan event, we can roll back the tape and watch how the order book evolved—which levels were defended, which were abandoned. That insight informs our model adjustments and risk management.

We encourage every developer to start small—maybe just one pair, one snapshot per second—and expand over time. The data you accumulate will become your most valuable asset for strategy innovation.

How to Recover Real-Time Stock Snapshots After a Trading Halt

EmilyL — Wed, 22 Jul 2026 02:43:19 +0000

When you’re building a trading bot, it’s easy to focus on signal logic and forget that market data has state. Stocks get suspended, go into auctions, or resume trading after days of silence. If your system doesn’t explicitly handle these transitions, you’ll end up with corrupted snapshots and phantom signals. Today, I’ll walk through how our team solved this at a high-frequency prop shop, and give you a reusable pattern you can drop into your own pipeline.

Scenario: The resumption that wasn’t
Imagine your strategy holds a stock that’s halted for an acquisition announcement. During the halt, your local cache shows the last trade at $50. Two weeks later, the stock reopens at $62. If your code simply updates the cache with the first new tick it receives, it may average these values, generate a false breakout, or trip a circuit breaker. We learned this the hard way when a misprocessed resumption triggered a $200k accidental unwind.

Why naive approaches fail
A quick comparison of common data access methods:

HTTP polling – You get the latest price every few seconds, but you don’t know if the stock is halted or if the API is returning a cached response. Timestamp often remains unchanged during a halt, so you can’t detect the resume event quickly.
Web scraping – Unreliable for state; you’ll be guessing based on screen text.
Broker SDKs – Sometimes include status, but it may be on a different stream or require additional parsing, adding latency and complexity.

We needed an event-driven solution that delivers the trading status with the tick.

Enter tick-level state fields
This is where a professional real-time WebSocket API shines. With AllTick’s tick stream, every message includes a status indicator (e.g., “TRADING”, “HALTED”). That means you can build a state machine entirely within your message handler, no external calls necessary.

Validation fields you must check
When a halt ends, don’t trust the first tick blindly. Verify these fields:

Data Field	What It Tells You
Instrument status	Must equal “TRADING”
Timestamp	Must be newer than the halt start time
Last price	The actual post-resumption transaction price
Volume	Should be > 0 to confirm real trade
Bid/ask depths	Must be present and non-stale

We use a simple last_valid_ts variable per symbol. Every incoming tick must have timestamp > last_valid_ts to be accepted. This prevents old ticks from reordering and overriding fresh data.

The recovery flow
Here’s the step-by-step state machine we implemented:

Normal trading: Continuously update snapshot cache.
Halt detected: Freeze cache, save pre-halt snapshot as a separate object.
Resumption signal received: Set state to WAITING_FIRST_TICK.
First tick validation: Check timestamp > halt_ts, volume > 0, price ≠ null.
Cache unfreeze: Overwrite snapshot with validated tick and notify downstream modules (K-lines, risk, etc.).

This separation between cache and state keeps your data lineage clean and lets you replay events for debugging.

Code example
Here’s a minimal WebSocket subscriber. In real use, you’d expand the on_message function with the state checks discussed above.

import websocket
import json

# Real-time tick subscription endpoint
api_doc = "https://apis.alltick.co/websocket-api/stock-websocket-interface-api/transaction-quote-subscription"

def on_message(ws, message):
    data = json.loads(message)
    timestamp = data.get("timestamp")
    if timestamp:
        # Placeholder for state validation and snapshot update
        print("update market snapshot:", data)

ws = websocket.WebSocketApp(
    api_doc,
    on_message=on_message
)
ws.run_forever()

To productionize this, wrap the logic in a class that manages per-symbol state and timestamps, perhaps backed by Redis for persistence across restarts.

Practical recommendations

Always run a pre-validation step before sending data to your strategy. If the status isn’t “TRADING,” skip the tick or route it to a monitoring queue.
Keep the pre-halt snapshot for audit; it’s invaluable for post-trade analysis.
Test your recovery with historical halt/resumption events. Replay them against your handler to catch edge cases early.

Remember: tick data is not a continuous film; it’s a series of snapshots with occasional intermissions. Respect those breaks, and your algorithms will thank you.

Building a Real‑Time Gold Breakout Detector for Cross‑Border Portfolios — A FinTech Lead’s Workflow

EmilyL — Thu, 16 Jul 2026 03:14:47 +0000

When you build data pipelines for cross‑border investors, “close enough” doesn’t cut it. A gold breakout that fires 20 seconds late, or mislabels a tiny spike as a trend change, can ripple into hedging errors across currencies. In this tutorial, I’ll share the real‑time detection workflow we use internally, compare the data access patterns we evaluated, and show you how to implement a state‑aware breakout monitor with WebSockets.

A Cross‑Border Investor’s Monitoring Nightmare

Picture a trader juggling gold exposure in USD, EUR, and JPY. Her system polls a REST endpoint every 10 seconds. Gold shoots through resistance at 2,080, pulls back, and the alert arrives when the price is already 2,075. She hedges on stale information. We fixed this by treating the breakout as a continuous state machine fed by tick‑level data, not a discrete threshold check.

Polling vs Streaming: What Worked for Us

We benchmarked three approaches before settling on a streaming architecture:

Approach	Latency	Granularity	Suitability for Breakout Detection
REST polling (5–10s)	High	Snapshot only	Misses intra‑interval spikes
Delayed WebSocket feeds	Medium	Batched	Unreliable timestamps break state machine
Native tick‑by‑tick WebSocket	Ultra‑low	Every tick	Ideal — captures the full micro‑structure

For gold, where moves accelerate around news and fixings, only the tick‑by‑tick stream gave us the temporal resolution to confidently distinguish a breakout from noise.

Why We Chose AllTick API for Precious Metals Streaming

During our comparison, we connected to multiple providers. The AllTick API stood out for its clean, unauthenticated WebSocket handshake (great for prototyping), low latency tick delivery for gold, and straightforward JSON subscription model. It became our backbone for precious metals real‑time data.

Implementing the Breakout Monitor

Let’s walk through the core components.

Step 1: Subscribe to Gold Ticks

We open a WebSocket connection and subscribe to GOLD. The tick objects arrive with price, timestamp, and volume.

# WebSocket real-time quote subscription example
url = "https://apis.alltick.co/websocket-api/stock-websocket-interface-api/transaction-quote-subscription"

symbol = "GOLD"

subscribe_data = {
    "action": "subscribe",
    "symbol": symbol,
    "type": "tick"
}

print("Starting live gold tick listener")

Step 2: Stateful Breakout Logic

A simple condition isn’t enough. We use a multi‑stage check that considers price level, change rate, and dwell time.

if current_price > resistance_price:
    breakout_status = "Breakout Under Observation"

    if price_change_rate > threshold:
        breakout_status = "Confirmed Breakout"
else:
    breakout_status = "No Breakout"

In production, we wrap this in a class that tracks consecutive ticks above resistance. If the count hits a minimum (e.g., 5) and the rolling volatility expands, we promote the status. Otherwise, we suppress the alert. This dramatically cuts false positives.

Step 3: Enhance with Historical Ranges

We pull historical daily candles to compute support and resistance zones. When the live price enters a zone that previously saw multiple failures or breakouts, we adjust the required confirmation time and rate thresholds dynamically. This makes the system sensitive to high‑value levels and relaxed in messy ranges.

Step 4: Production Hardening

Timestamp hygiene: All ticks are stamped in UTC milliseconds upon arrival and ordered by sequence number. Out‑of‑order ticks trigger a resync.
Reconnection strategy: Our WebSocket client auto‑reconnects and compares the last processed timestamp with the first tick of the new stream. Any gap is logged and a REST snapshot is used to realign.
Heartbeat watch: Missing ticks for more than a configurable window flags the status as “stale,” preventing downstream consumers from acting on dead data.

Key Takeaways for Developers

Real‑time data is not a luxury for cross‑border gold monitoring; it’s the foundation that prevents entire categories of false breakouts.
A small state machine with dwell time and volatility checks outperforms complex ML models when explainability and speed matter.
Invest time in stream resilience early: reconnection, ordering, and heartbeats will save you from 3‑AM wake‑up calls.

Link to the API we used: AllTick API documentation — explore the WebSocket interface that powers our gold monitor.

Ensuring Timestamp Continuity in Historical Stock Replay — A Quant Architect’s Cost-Efficient Playbook

EmilyL — Wed, 15 Jul 2026 03:11:00 +0000

I work as a strategy chief architect for quantitative hedge funds and private equity managers. My title often drifts into “profit curve optimizer” because my mandate is brutally simple: make the backtest equity curve match live trading, and do it without blowing the tech budget. In this post, I’ll share the exact timestamp hygiene protocol I use to banish replay distortions — a protocol that delivers institutional-grade reliability on a shoestring.

Understanding the Client’s Core Need: Reliable Time on a Lean Budget

Quant fund managers and private equity investors don’t want another dashboard. They want replay veracity. They need to know that if a strategy shows a 2.5 Sharpe in simulation, it won’t embarrass them with a 0.3 in production. And critically, they want this assurance without paying for atomic clocks or proprietary timestamping hardware. Cost-effectiveness is the lens through which every engineering decision is judged.

Where Replay Goes Wrong: Advisor-Side Pain Points

When I first wired a stock quote WebSocket into a replay store, I assumed data completeness equaled data correctness. Reality taught me otherwise. The pipeline introduced three classes of timestamp anomalies that I now actively guard against:

Out‑of‑order delivery: network jitter reversing the true trade sequence.
Granularity clash: some vendors emit second‑level stamps, others millisecond.
Gap misinterpretation: a market closure being replayed as an instantaneous price jump, shredding short‑term indicators.

These issues don’t just annoy quants — they erode the advisor’s faith in their own research, leading to delayed launches and missed alpha windows.

Data-Backed Solution: A Lightweight Temporal Governance Model

I introduced a three‑tier timestamp policy that requires zero additional software cost:

Time Field	Function
market_time	The definitive sort key for all replay
receive_time	Latency monitoring and anomaly alerts
store_time	Root‑cause analysis for data pipeline issues

A fast pre‑replay validator enforces three rules: timestamps must be strictly increasing (with configurable tolerance for bursts), gaps wider than a dynamic threshold are flagged, and repeated timestamps are cross‑checked with price and volume to distinguish authentic fills from duplicate packets. Across a dozen funds I’ve supported, this simple scheme has brought the backtest‑to‑live tracking error down to less than 15 bps on daily strategies.

Service Upgrade: Aligning Live Capture and Replay Clocks

The last mile was removing local server time from the equation entirely. I now ingest only the exchange‑provided timestamp from the real‑time feed. For example, when leveraging AllTick API for tick capture, I store the timestamp as the golden field:

# WebSocket subscription for real-time market data
url = "https://apis.alltick.co/websocket-api/stock-websocket-interface-api/transaction-quote-subscription"
tick_time = data["timestamp"]

# Persist using the exchange market time
save_tick({
    "time": tick_time,
    "price": data["price"],
    "volume": data["volume"]
})

Once your live and historical pipelines share the exact same time anchor, you eliminate the silent mismatch that poisons countless backtests. Top it off with sane exception handling — gap journals, fingerprint‑based deduplication, pre‑built time indices — and your replay module becomes a trusted foundation instead of a source of doubt.

I’ve learned that in systematic trading, expensive hardware rarely solves what disciplined timestamp logic can fix for free. Get your time axis right, and the rest of the strategy stack suddenly behaves.

How to Build Unbroken Candlestick Charts for US Stocks: A Three-Layer Session Tagging Approach

EmilyL — Tue, 07 Jul 2026 06:02:08 +0000

The Problem: Candlestick Gaps That Confuse Users and Algorithms

When our team integrated US stock market data into a platform originally built for cryptocurrency traders, we ran into a persistent issue: candlestick charts exhibited ugly jumps around market open and close. Pre-market price moves would disappear from the visual timeline, and low-volume after-hours prints would create phantom volume spikes. The root cause? Most raw data feeds dump all ticks into a single undifferentiated stream, ignoring the fact that US equities trade in three distinct sessions — pre-market (4:00–9:30 ET), regular (9:30–16:00), and after-hours (16:00–20:00). Our users, many of whom are professional advisors and algo traders, demanded a chart that was visually continuous without sacrificing logical accuracy.

Traditional Pains: When Advisors Manually Strip Session Noise

Before we shipped our solution, we spent hours interviewing power users. A recurring complaint: they were forced to manually filter out non-regular-hours data before running technical analysis, because indicators like VWAP and RSI would be skewed by stray trades in thin markets. This manual step was error-prone and impossible to automate at scale. They needed the platform to handle session semantics transparently, leaving them to focus on strategy.

The Data Architecture That Made Continuity Possible

We tackled this by designing a three-tier tick processing pipeline:

Raw Tick Ingestion – Using real-time WebSocket streams (e.g., AllTick), we capture every transaction with microsecond precision. No filtering, no judgment — just collection.
Normalization & Session Labeling – All timestamps are converted to US Eastern Time. A simple time-window function assigns each tick a session label (pre_market, regular, after_hours). Outlier filtering happens here as well.
Session-Aware OHLC Aggregation – When building K-lines, we partition ticks by session and time bucket. Crucially, we never merge volumes across session boundaries, which eliminates the phantom spike problem entirely. Bars are generated only where actual trades exist — we never fabricate "filler" candles.

# WebSocket tick ingestion with session labeling downstream
import websocket
import json

url = "wss://stream.alltick.co/v1/stock/realtime"

def on_message(ws, message):
    data = json.loads(message)
    # In production, session label and aggregation logic are applied here
    print(data)

def on_open(ws):
    sub_msg = {
        "action": "subscribe",
        "channel": "transaction_quote",
        "symbols": ["AAPL"]
    }
    ws.send(json.dumps(sub_msg))

ws = websocket.WebSocketApp(
    url,
    on_message=on_message,
    on_open=on_open
)
ws.run_forever()

The result: a visually seamless candlestick chart where each segment knows whether it belongs to pre-market, regular, or after-hours trading. Algorithms can optionally filter by session, but the default view is free of artificial gaps.

Service Upgrade: Making Session Intelligence a Platform Feature

We subsequently wrapped this pipeline into a dedicated market-data microservice. Product teams can now request session-aware candlestick series via a single API parameter. This has significantly reduced onboarding time for new chart features and allowed our quant community to run session-specific backtests without extra data wrangling. What started as a chart glitch turned into a key differentiator.

How to Reliably Get XAUUSD Daily OHLC Data with Python (and Avoid Common Traps)

EmilyL — Wed, 24 Jun 2026 06:15:17 +0000

In this tutorial, I’m going to share a practical method to fetch London gold (XAU/USD) daily bars using Python, based on what I’ve learned as a financial data analyst. If you’ve ever tried plugging different metals APIs into your backtest and gotten wildly different results, this one’s for you.

Why Daily Data Matters in Your Quant Workflow

I treat daily bars as the directional compass for my gold strategies. They don’t provide entry signals, but they determine whether I should be looking for longs or shorts. If the daily data is polluted with misaligned opens or missing days, my entire signal generation layer becomes unreliable. Gold, with its strong technical structure, demands a particularly clean dataset.

Common Pitfalls with Precious Metals APIs

The two issues I see most often are time misalignment and data gaps. Some APIs define the daily bar by the UTC calendar day, while others use the New York trading close. This creates inconsistencies in high, low, and close values. Additionally, missing dates in the historical record can make volatility calculations appear artificially low unless properly handled.

The Data Structure You Should Expect

Here’s the standard OHLC format I work with:

Field	Meaning
open	Opening price
high	Highest price
low	Lowest price
close	Closing price
volume	Volume
timestamp	Time

For gold, I always scrutinize high and low because they capture the extreme points where stop-loss clusters tend to sit. Ignoring these wicks can lead to missing critical reversal signals.

Step-by-Step: Fetching and Cleaning with Python

The code below uses an API that returns UTC-based timestamps consistently. In my experience, picking a source like AllTick that enforces this standard from the start saves a ton of alignment effort.

import requests
import pandas as pd

# API endpoint for historical klines
url = "https://api.alltick.co/v1/klines"

# Request parameters for gold daily data
params = {
    "symbol": "XAUUSD",
    "interval": "1d",
    "limit": 500
}

resp = requests.get(url, params=params)
data = resp.json()

# Convert the JSON data into a pandas DataFrame
df = pd.DataFrame(data["data"])

# Convert the timestamp from milliseconds to datetime
df["timestamp"] = pd.to_datetime(df["timestamp"], unit="ms")

# Ensure data is sorted by time
df = df.sort_values("timestamp")

print(df.head())

Once I have this DataFrame, I run a gap check: I create a full date index using pd.date_range, reindex the data, and forward-fill missing bars while marking them with a flag. This approach keeps the temporal continuity intact without inventing phantom price action.

Integrating into Your Strategy

I’ve used this exact data prep flow for moving average crossovers, Donchian breakouts, and volatility filter modules. The outcome is always the same — more stable signals and fewer false positives during backtesting. Clean daily data isn’t glamorous, but it’s the cheapest performance boost you can give your gold trading system.

Why Your Real-Time BTC/USDT Tick Data Might Be Feeding You Incomplete Stories

EmilyL — Wed, 17 Jun 2026 06:29:32 +0000

Hey devs! I want to talk about something that doesn’t get enough attention in the crypto dev space: the integrity of your tick-by-tick trade data stream. Not the API, not the WebSocket library, but what happens after the bytes arrive in your application.

Defining the Real Requirement

When you connect to a BTC/USDT trade stream, each message contains the fundamentals:

Field	Meaning
price	Trade price
volume	Trade quantity
timestamp	Execution time
side	Taker side
trade_id	Unique ID

The developer’s mission is not just to parse these fields, but to guarantee that the sequence of trades your system processes matches the sequence that actually happened on the exchange. That’s a much harder problem. Missing trades, misordered events, or timestamp mismatches all corrupt the market’s ground truth before your logic even sees it.

The Common Pitfall: Underestimating the Pipeline

In many projects, the data ingestion layer is treated as a trivial step — “just use a WebSocket client, and you’re done.” But real-world BTC/USDT feeds, especially during volatility, push data at extremely high rates. If your on_message callback takes even a little too long, the internal buffer can overflow, dropping trades without any error. Combine that with network reordering and timezone mixing, and you’ve got three silent killers of data quality.

Establishing a Reliable Data Source

Using a focused real-time data provider helps. For instance, I’ve worked with AllTick’s API, which delivers structured tick data via WebSocket. A minimal connection looks like this:

import websocket
import json

url = "wss://stream.alltick.co/ws/v1?token=demo"

def on_message(ws, message):
    data = json.loads(message)

    trade = {
        "symbol": data.get("symbol"),
        "price": float(data.get("price", 0)),
        "volume": float(data.get("volume", 0)),
        "timestamp": data.get("ts")
    }

    print(trade)

def on_open(ws):
    msg = {
        "action": "subscribe",
        "params": {
            "symbol": "BTCUSDT",
            "channel": "trade"
        }
    }
    ws.send(json.dumps(msg))

ws = websocket.WebSocketApp(url, on_open=on_open, on_message=on_message)
ws.run_forever()

That’s the easy part. Now let’s talk about making it robust.

Upgrading Your Data Service Layer

I implemented three key improvements that turned my brittle prototype into a reliable system:

Queue-Based Decoupling: I moved all processing out of the WebSocket callback. Raw messages go into a queue, and a separate thread pool handles the rest. This keeps the receive path fast and prevents buffer overruns.
UTC Normalization: I enforce strict UTC usage from the moment data is parsed. Any interaction with local time is explicit and isolated. This avoids edge-case bugs around hour boundaries.
Reordering Buffer: A small sliding window sorts incoming trades by timestamp. This corrects the occasional out-of-order delivery caused by network paths without adding perceptible delay.

Focus on Stability, Not Just Speed

What I learned over time is that a rock-solid, consistent tick feed is a superpower. The flashy part of quant work is the strategy, but the foundation is the data. If your data stream has gaps, your strategy is making decisions on a distorted view of the market. So before optimizing your algorithm’s next microsecond, spend time making sure every single trade is actually reaching your logic, in the right order, with the right time. That’s the kind of engineering that pays off silently but massively.

Happy coding, and may your ticks always be in order!

How Do You Handle WebSocket Drops and Missing Ticks in US Stock Data? Here’s My Approach

EmilyL — Wed, 10 Jun 2026 02:58:54 +0000

If you’ve ever run a tick-based backtest and ended up with results that just didn’t add up, the culprit might be hiding in your data pipeline — specifically, in those few seconds when your WebSocket feed went silent without you noticing.

As a financial data analyst working with US equities, I’ve been through this more times than I’d like. Today, I’ll walk through the reconnection and backfill mechanism I built to ensure my tick stream remains complete and correctly ordered, even when the network misbehaves.

The Problem: Real-Time Feeds Aren’t “Set and Forget”

WebSocket connections for live tick data are long-lived, which makes them susceptible to network hiccups, server-side maintenance, and local resource limits. A common pitfall is assuming that a library’s auto-reconnect feature solves everything. It doesn’t — most auto-reconnects only resume from “now,” leaving a gap in your time series.

That gap might contain the very trade that triggered your entry signal or the quote that would have prevented a false breakout. So we need two things: automatic reconnection with full state recovery and automatic backfill of missing data.

Reconnection That Remembers

My reconnection strategy is built around these principles:

Heartbeat watchdog: I set an application-level timer. If no message (including ping/pong) is received within a threshold (e.g., 15 seconds), I forcibly close the socket and initiate reconnection. This proactive detection beats waiting for OS-level timeouts.
Persist subscription context: I keep the list of subscribed tickers in memory. Upon reconnection, the system resubscribes to every ticker automatically — no missed symbols, no manual re-entry.
Backoff and circuit breaker: Continuous rapid reconnects can get your IP blacklisted. I use exponential backoff between attempts and a cap on consecutive failures, after which the system pauses and alerts.
Handoff to backfill: Once reconnected, a flag is set along with the timestamp of the last successfully received tick. This tells the backfill module exactly where the hole begins.

Filling the Hole with Historical Data

The backfill process is essentially a time-range query against a historical API:

Watermark tracking: During normal streaming, I continuously update a last_tick_time variable. In production, I also persist it to a fast store like Redis.
Freeze on disconnect: When the feed drops, last_tick_time stops advancing. It now defines the start of the missing period.
Retrieve missing ticks: After reconnection, a GET request is sent to the historical tick endpoint with start_time set to the frozen watermark.
Merge in order: The returned ticks and the live ticks both enter a time-sorted buffer. Downstream consumers always process events in strict chronological order.

In my setup, I rely on data platforms like AllTick that offer aligned real-time and historical schemas. This means the backfilled ticks can be inserted directly into the same processing pipeline without any translation layer.

import websocket
import json
import time
import requests

# Last tick timestamp stored locally
last_tick_time = "2026-06-05T10:15:00Z"

def on_message(ws, message):
    data = json.loads(message)
    global last_tick_time
    last_tick_time = data["timestamp"]
    print(data)

# WebSocket reconnection
def reconnect():
    ws = websocket.WebSocketApp(
        "wss://apis.alltick.co/stock/ws",
        on_message=on_message
    )
    ws.run_forever()

reconnect()

# Backfill missing historical ticks
def fetch_missing_data(start_time):
    url = f"https://apis.alltick.co/stock/api/history?start_time={start_time}"
    resp = requests.get(url)
    ticks = resp.json()
    for tick in ticks:
        print(tick)
    # Update the last tick timestamp
    global last_tick_time
    if ticks:
        last_tick_time = ticks[-1]["timestamp"]

fetch_missing_data(last_tick_time)

Practical Tips for Stability

Sorted merge buffer: Use a priority queue keyed by timestamp to merge live and backfilled data. This eliminates any race-condition-induced ordering issues.
Fine-grained watermarks: Maintain a separate last_tick_time per symbol. This scopes backfill requests to only the affected tickers, reducing load.
Comprehensive logging: Record every disconnection event — when it happened, how long it lasted, and how many ticks were backfilled. This data is gold when auditing strategy performance or evaluating provider reliability.
Staggered subscription: After reconnecting, subscribe to symbols in batches with short delays to keep your system’s resource usage steady.

Wrapping Up

Solid data engineering doesn’t just make your pipeline faster — it makes your analysis trustworthy. By combining fast heartbeat detection, stateful reconnection, and precise historical backfill, you can turn an unreliable streaming feed into a dependable foundation for quantitative research. Give it a try in your next project, and you’ll likely spend a lot less time second-guessing your backtest results.

How to Aggregate Real‑Time Stock Ticks into 1‑Minute Candles Using WebSocket and Pandas

EmilyL — Tue, 09 Jun 2026 03:39:13 +0000

Have you ever tried plotting a real‑time intraday chart only to watch the line fracture into disconnected segments? The culprit is almost always missing data for certain minutes. In this tutorial, I’ll walk you through a pattern that turns a turbulent WebSocket tick stream into a smooth, reliable 1‑minute series—perfect for financial dashboards, trading bots, or just satisfying your own data curiosity.

Why minutes go missing

When you query minute‑level endpoints, the server usually returns only the minutes that had trades. If a stock sits idle for sixty seconds, that slot simply doesn’t appear. Front‑end libraries connect the dots they have, leaving an ugly jump. The solution? Accept raw trades and build the minute bars yourself, ensuring every minute gets a value.

The fields you absolutely need

Before we code, let’s lock down the schema for a minute bar:

timestamp aligned to the minute floor.
price using the last trade of that minute.
volume (optional but helpful) aggregated over the window.

Getting the tick stream

I’m using a WebSocket connection to AllTick in this example because it delivers the essential fields in a straightforward JSON format. Here’s the basic subscriber:

import websocket
import json

def on_message(ws, message):
    data = json.loads(message)
    # Display each incoming tick
    print(f"Time: {data['time']}, Price: {data['price']}, Volume: {data['volume']}")

def on_open(ws):
    subscribe_data = {
        "action": "subscribe",
        "symbol": "AAPL"
    }
    ws.send(json.dumps(subscribe_data))

ws = websocket.WebSocketApp(
    "wss://api.alltick.co/stock",
    on_message=on_message,
    on_open=on_open
)

ws.run_forever()

In a real application, you’d push these ticks into a buffer or a lightweight time‑series database instead of just printing them.

Aggregating to one‑minute candles

pandas makes resampling almost trivial. We parse the timestamps, set them as the index, and call resample('1min'). The last() method grabs the final price of each minute, and ffill() propagates the last known price through any empty minutes.

import pandas as pd

df['time'] = pd.to_datetime(df['time'])
df.set_index('time', inplace=True)

# Resample ticks to 1-minute bars, forward-filling any gaps
df_1min = df['price'].resample('1min').last().ffill()

After this step, df_1min is a Series with a continuous time index—no more holes.

Plotting the minute chart

With a clean sequence, we can plot using plotly for a nice interactive experience:

import plotly.graph_objects as go

fig = go.Figure()
fig.add_trace(
    go.Scatter(
        x=df_1min.index,
        y=df_1min.values,
        mode='lines',
        name='Price'
    )
)
fig.show()

Feel free to add a bar trace for volume to make the chart even richer.

Why this matters beyond the tutorial

Steady, gap‑free minute data is the foundation of many academic studies in finance, such as analyzing intraday volatility patterns or testing algorithmic trading strategies. By learning to control the aggregation yourself, you’re not just fixing a chart—you’re gaining a skill that opens the door to serious quantitative work. Next time your intraday line misbehaves, you’ll know exactly where to look.

Monitoring Gold Price API Latency: A Practical WebSocket Check with Distribution Analysis

EmilyL — Wed, 03 Jun 2026 06:13:27 +0000

We’re a team of individual traders who write our own tools, and recently we tackled a very practical challenge: how to objectively measure and evaluate the timestamp delay from a gold price API’s real-time push. When gold market volatility spikes, a few milliseconds of unexpected jitter can throw off our automated strategies. So we developed a straightforward, distribution-based monitoring pattern that we’d like to share.

Understanding the Delay Components

Every real-time gold tick contains a server-side timestamp. When we calculate the difference between that and our local clock, we get the end-to-end latency. The key is to realize this latency is the sum of many small parts: exchange processing, data aggregation, network travel, and local parsing. Looking at a single outlier tells you nothing useful. The requirement is to observe the whole distribution.

Key Time Points We Instrument

We instrument four logical timestamps in our monitoring:

Source generation time: timestamp on the quote.
Edge out time: when the feed server sends the packet.
Local receive time: when our OS captures the frame.
Parse complete time: when the tick is available to the trading algorithm.

With these points, diagnosing a lag spike becomes straightforward. If the delta between receive and parse grows, our local consumer is likely blocking. This structured approach solved a lot of our debugging pain.

Typical Latency Ranges We See

Based on our logs in varied environments, gold price push delays generally fall into these buckets:

Network path	Typical latency
Same data center / region	10ms – 50ms
Cross-region internet	50ms – 200ms
Periods of network unrest	200ms – 500ms

We don’t panic at the high end of the range unless it becomes the norm. The data point that concerns us most is distribution variance; a steady stream is what we need.

A Quick Python Script for Latency Distribution

Here’s the small script we use to capture latency samples. We connect to a gold feed’s WebSocket endpoint (for instance, using the AllTick API, which serves reliable structured timestamps) and simply log the differences.

import websocket
import json
import time

def on_message(ws, message):
    data = json.loads(message)
    # Grab the server timestamp from payload
    server_ts = data.get("ts")
    local_ts = int(time.time() * 1000)

    diff = local_ts - server_ts
    print("delay(ms):", diff, data)

ws = websocket.WebSocketApp(
    "wss://stream.alltick.co",
    on_message=on_message
)

ws.run_forever()

Collecting the output and plotting it in a histogram gives us a latency “fingerprint” of the gold API for that session. It’s a huge improvement over staring at raw numbers.

Prioritizing Stability Over Minimum Latency

The biggest takeaway from our monitoring practice is that a stable and predictable delay profile trumps an ultra-low but erratic one. When we know the delay distribution is consistent, we can confidently adjust our strategy parameters. Our daily focus has shifted from chasing the fastest tick to building a robust, observable pipeline that keeps time reliably.

Python Tip: Distinguishing Pre-market, Regular, and After-hours Ticks from a WebSocket Stream

EmilyL — Tue, 02 Jun 2026 03:43:33 +0000

Ever received a WebSocket tick stream for US stocks and wondered why your indicators behave oddly outside regular hours? The raw data doesn’t tell you which session a trade belongs to, but identifying the session is crucial for signal quality. Here’s a clean, no-dependency-heavy way to do it in Python.

Quick Session Reference

Session	US Eastern Time	Data Characteristics
Pre-market	04:00-09:30	Sparse trades, choppy moves
Regular hours	09:30-16:00	Dense liquidity, smooth price action
After-hours	16:00-20:00	Volatility often triggered by news

Method 1: Timestamp Conversion

Almost every API sends a UTC timestamp. Convert it to US/Eastern and classify.

from datetime import datetime
import pytz

# US Eastern timezone
et = pytz.timezone('US/Eastern')

def get_session(ts):
    t = datetime.fromtimestamp(ts, et)
    # Check pre-market window
    if t.hour < 9 or (t.hour == 9 and t.minute < 30):
        return "pre"
    # Regular session
    if t.hour < 16:
        return "regular"
    # After-hours
    return "after"

Method 2: Use a Session Status Field

If your provider sends a field like sessionType, you can skip the timezone math. Just make sure to test edge cases at session boundaries.

Live Integration Example

Using a WebSocket feed (like AllTick’s market data stream) that includes a timestamp, I label ticks on the fly.

import websocket
import json
from datetime import datetime
import pytz

# US Eastern timezone
et = pytz.timezone('US/Eastern')

def session(ts):
    t = datetime.fromtimestamp(ts, et)
    if t.hour < 9 or (t.hour == 9 and t.minute < 30):
        return "pre"
    elif t.hour < 16:
        return "regular"
    else:
        return "after"

def on_message(ws, message):
    data = json.loads(message)
    s = session(data["timestamp"])
    print(f"{data['symbol']} | {s} | {data['price']} | {data['volume']}")

# Open WebSocket connection
ws = websocket.WebSocketApp("wss://ws.alltick.co/stock", on_message=on_message)
ws.run_forever()

Efficiency takeaway: Doing session classification at ingestion keeps the rest of your pipeline clean. Each downstream module simply filters by session == "regular" and ignores the noise. It’s a tiny compute cost that prevents huge modeling headaches down the line. Hope this helps your next real-time project!

How to Batch Subscribe to 5000+ A-Share Real-Time Quotes with a Single WebSocket (Python)

EmilyL — Wed, 27 May 2026 06:49:11 +0000

When I first tackled real-time market data for the entire A-share market, I did what many devs do: I wrote a loop that polled a REST endpoint for each stock. It was slow. It was fragile. And it was constantly hitting rate limits.

The solution? Tear down the polling loop and put up a WebSocket subscription. This post walks through why you should too, and shows you the code to get started.

The Problem with Polling 5000+ Instruments

Polling is a synchronous ask-reply model. You request, you wait, you get a response. When multiplied by thousands of stocks and the high tick frequency of A-shares (hundreds of trades per second during peak), you end up with a stream of delayed snapshots, not a real-time feed.

WebSocket push turns the model on its head. The server sends you data only when something happens, over a persistent connection. This means:

Millisecond-level latency
No wasted requests or rate limits
One TCP connection regardless of how many symbols you track

Two Common Subscription Message Formats

Different APIs accept slightly different shapes. Usually it's either:

Format	Sample	Use Case
Array	`["000001","000002","600036"]`	Great for dynamic watchlists
String	`"000001,000002,600036"`	Compact, easy for URL params

Some providers also let you use a wildcard to subscribe to all listed A-shares — amazing for scanners, but you’ll need to get the appropriate access tier.

Code: One WebSocket, Ten Stocks, Zero Polling

Here’s a basic Python script using websocket-client. It connects to a push endpoint (think AllTick-style APIs) and subscribes to a batch of symbols.

import websocket
import json

# WebSocket endpoint that streams real-time trade ticks
url = "wss://apis.alltick.co/websocket-api/stock-websocket-interface-api/transaction-quote-subscription"

def on_message(ws, message):
    # Decode the incoming JSON
    data = json.loads(message)
    # Iterate over all tick updates
    for tick in data.get("ticks", []):
        print(f"Code:{tick['code']} Price:{tick['price']} Time:{tick['time']}")

def on_open(ws):
    # Batch subscription request for 10 A-shares
    sub_msg = {
        "action": "subscribe",
        "symbols": ["000001", "000002", "600036", "600519", "000858",
                    "002415", "300750", "601318", "000333", "002594"]
    }
    ws.send(json.dumps(sub_msg))

ws = websocket.WebSocketApp(url, on_message=on_message)
ws.on_open = on_open
ws.run_forever()

Run this, and every time any of those stocks trades, you’ll see the price, code, and timestamp instantly.

Surviving the Data Firehose

Once you scale to the full market, you might see hundreds of ticks per second. Here’s how I handle it in production:

Sharded processing: Hash by stock code and feed separate worker queues.
Snapshot cache: Keep a dict of latest prices for O(1) read access.
Batch DB inserts: Accumulate ticks and flush in micro-batches (e.g., 100 records or every 200ms).
Threshold-based UI updates: Only push price changes larger than a set tick size to front-end clients.

Preparing for All-Market Subscription

If you’re going to scan the entire market, make sure:

Your network and CPU can handle peak loads (I’ve logged ~300–400 ticks/sec during openings).
You introduce a buffer (like Redis Streams or Kafka) when processing can’t keep up.
You enable server-side filters that send ticks only for stocks with actual trades.

Wrapping Up

Stop polling. Start pushing. A single WebSocket with batch subscription is the most efficient way I’ve found to get full A-share tick data. When evaluating a provider, focus on latency, batch support, and stability. Then start small, stress-test, and scale gradually.

Your future self — and your servers — will thank you.