DEV Community: Emily

How to Stop Your Trading Bot’s Multi-Timeframe Signals from Fighting Each Other

Emily — Tue, 14 Jul 2026 08:06:27 +0000

The real-world user requirement You’re developing the signal engine for a smart investment app, and the brief from the product team is straightforward: “Make the recommendations feel less schizophrenic.” Users are uninstalling because a daily-bullish stance keeps getting interrupted by minute-level sell alerts. As the engineer who also thinks like a PM, you recognize the requirement beneath the complaint — the client doesn’t need more indicators, they need cross-period consensus.

The pain point: ungoverned multi-period signals In your initial architecture, you probably treated the daily, hourly, and minute charts as independent feature sets. A breakout on the minute chart fires a buy; a dark cloud cover on the hourly fires a sell. When these signals coexist without a governing hierarchy, your bot broadcasts whiplash. The deep issue is that lower timeframes contain a high ratio of random noise to actionable information. Giving them equal weight poisons the user experience and leads to false entries, premature exits, and shattered trust in the robo-advisor.

Designing a top-down validation filter The solution is to implement a signal arbitration layer that enforces a strict sequence: the higher timeframe authorizes, the lower timeframe executes.

Timeframe Arbitration Role
Daily Defines the allowed directional bias
Hourly Confirms that retracements are orderly
Minute Generates entry signals that must pass the above checks
You can wrap this into a lightweight validator. Before any alert is pushed to the user, the validator checks: (1) Does the daily trend permit this trade? (2) Is the hourly structure supportive (e.g., pullback not violating key level)? Only if both are true does the minute-level trigger get approved. This filter slashes false positives without redesigning your core strategies.

Unifying the data source so periods actually match A validator is useless if the daily bar and the minute bar come from different data pipelines. You might have experienced a scenario where a “breakout” on the minute chart appears a few seconds before the daily bar officially closed, creating a signal that doesn’t exist in reality. To eliminate timestamp drift, you need to rebuild all candles from the same raw tick flow.

You can use a real-time market data API that offers WebSocket tick streaming — like AllTick API. Aggregate candles in memory, and then your daily, hourly, and minute bars will share the exact same price origin.

import websocket
import json

Ingest real-time ticks to ensure synchronized multi-timeframe candles

url = "wss://quote.alltick.co/socket"

def on_message(ws, message):
# Raw tick data arrives here; aggregate into candles as needed
data = json.loads(message)
print(data)

ws = websocket.WebSocketApp(
url,
on_message=on_message
)

ws.run_forever()
With this approach, cross-timeframe validation runs on data that is genuinely aligned, making the arbitration logic dependable.

The measurable upgrade in advisory quality After deploying the arbitration layer and tick-sourced candles, your bot’s output stabilizes. The number of conflicting daily notifications drops, and the signals that do reach users carry the weight of multiple timeframes agreeing. Internally, you see improved user retention and fewer support escalations citing “bad calls.” The upgrade isn’t a new feature — it’s making the existing ones finally act in concert.

How Our Algorithmic Execution Team Repairs Missing Crypto K-lines with Tick Replay

Emily — Thu, 09 Jul 2026 06:16:26 +0000

When your execution algorithm relies on precise historical patterns, nothing stings more than discovering your backtest was trained on incomplete data. In our team, we’ve encountered numerous cases where a seemingly profitable strategy collapsed the moment we fed it a complete K-line series. In this guide, I’ll show you exactly how we detect and fix candlestick gaps in crypto data—without ever injecting synthetic prices.

Why Gaps Appear in a 24/7 Market

A 1-minute K-line should arrive every 60 seconds. In reality, gaps creep in through WebSocket disconnections, API rate limits, logger crashes, or timezone mismatches. The gap doesn’t mean the exchange paused; it means your recording pipeline missed the snapshot. If your execution logic trusts these gaps, it will make decisions on a distorted version of reality.

The Manual Grind We Left Behind

Early on, we checked for gaps by exporting data and visually scanning timestamps. This was slow, inconsistent, and soul-crushing. As execution engineers, our time should be spent optimizing latency and slicing algorithms, not playing data detective. The inefficiency was silently eroding our ability to ship robust strategies.

Our Automated Repair Workflow

We built a validation layer that automatically audits every historical dataset before it enters a backtest. The mandatory checks are summarized below:

Check	Goal
Timestamp interval consistency	Detect any gap by comparing actual diff to expected period
Duplicate timestamps	Avoid double-counting bars
OHLC price logic (High ≥ Low, etc.)	Reject corrupted rows
Volume behavior sanity	Confirm trades occurred where prices moved

If a gap is found, we skip interpolation entirely. Instead, we reconstruct the missing candle from tick data. The principle is straightforward: a candlestick is an aggregation of individual trades within a window. We collect all ticks in the missing interval and derive open (first trade), close (last trade), high, low, and summed volume. To have ticks always available, we maintain a continuous WebSocket feed.

import websocket
import json

url = "wss://quote.alltick.co/quote-stock-b-ws-api"

def on_message(ws, message):
    data = json.loads(message)
    print(data)

ws = websocket.WebSocketApp(
    url,
    on_message=on_message
)

ws.run_forever()

A provider like AllTick delivers the raw tick stream, which we persist with all timestamps normalized to UTC. This timestamp unification is critical—many apparent gaps are really timezone illusions, and converting everything to UTC instantly dissolves them.

How This Changed Our Daily Work

With the automated pipeline in place, we’ve completely eliminated manual data inspection. Our backtests now reflect genuine market conditions, and our execution algorithms behave identically from simulation to production. We’ve found that investing in data continuity often yields greater performance improvement than months of parameter optimization. If you’re building a trading bot, treat your data pipeline as the product’s foundation—it makes everything else stand.

Stop Backtesting, Start Streaming: Re-architecting Your Gold Trend Strategy for Tick Data

Emily — Tue, 30 Jun 2026 02:52:01 +0000

You've been there. You code up a sleek gold trend-following strategy in Python, run a vectorized backtest over five years of historical XAUUSD data, and the results look fantastic. The Sharpe ratio is solid, the drawdown is minimal. You're ready to go live. You hook it up to a WebSocket streaming real-time prices, and… it all falls apart. Your signals behave erratically, the latency makes your entries late, and your P&L curve looks nothing like the backtest.

The problem isn't your trading logic; it's that your system was architected for a static, perfect world, not the messy, real-time one. Let's debug and refactor this like the engineers we are.

The Bug in Your Paradigm: Trend is a State Machine, Not an Event

In backtesting, it's easy to treat a "trend" as a discrete event, like a boolean flag that flips on a golden cross. In the live, high-volatility world of gold, this is a critical bug. A price can break out, trigger your flag, and then immediately collapse in a deep, violent retracement.

You need to model a trend as a continuous, structural health state. Your code shouldn't check for a "start trend" event; it should constantly evaluate the health of the current market state. A healthy uptrend state requires three conditions to persist:

Intact Price Structure: A consistent series of higher highs and higher lows.
Confirming Momentum: The force behind the move is not diverging or fading.
Orderly Retracements: Pullbacks are shallow, indicating liquidity absorption, not a deeper reversal. When price keeps peaking higher but retracements are getting deeper, your state machine should already be transitioning to a "warning" or "neutral" state, long before a simple moving average crossover would catch up.

The Architectural Mismatch: Pull vs. Push Processing

This is the root cause. Your backtest runs a batch-processing (pull) job. It waits for a [candle_close] event, then queries a complete OHLC record and runs a function. The input data set is finite and static.

Live trading is a stream-processing (push) world. A firehose of tick data is pushed to your app via WebSocket. If your architecture is still waiting for a conceptual [candle_close] to do its work, you're processing stale data. You're calculating a signal based on a past event, and in gold, that micro-latency is costly. You must process the stream continuously.

Refactoring to a Three-Layered Pipeline

To fix this, we need to decouple the monolith with a clean, three-layered architecture:

Ingestion Layer: A thin client that handles the WebSocket connection, receives raw messages, and deserializes the JSON. It does zero business logic to stay fast.
Buffering Layer: The heart of the system. It's a fixed-length deque acting as a sliding window. It absorbs the high-frequency, jittery tick flow and provides a stable, contiguous data view to the layer above, smoothing out noise.
Calculation Layer: Your pure strategy logic. It gets a snapshot from the buffer, performs its calculations, and returns a state. It's completely isolated from I/O, making it deterministic and easily testable.

Here’s the refactored code example, demonstrating this pipeline for a simple dual-MA crossover on XAUUSD ticks:

import websocket
import json
from collections import deque

# Layer 2: Sliding window buffer
prices = deque(maxlen=20)

def signal():
    # Layer 3: Pure calculation logic
    if len(prices) < 20:
        return None  # Buffer not full

    short_ma = sum(list(prices)[-5:]) / 5
    long_ma = sum(prices) / 20

    if short_ma > long_ma:
        return "buy"
    elif short_ma < long_ma:
        return "sell"
    return "hold"

def on_message(ws, message):
    # Layer 1: Ingestion and injection
    data = json.loads(message)
    price = float(data["price"])
    prices.append(price) # Push to sliding window

    s = signal() # Check for new state
    if s:
        print(f"State: {s}, Price: {price}")

def on_open(ws):
    ws.send(json.dumps({
        "action": "subscribe",
        "symbol": "XAUUSD",
        "type": "tick",
        "id": 1
    }))

ws = websocket.WebSocketApp(
    "wss://api.alltick.co/ws", # Your reliable data source
    on_message=on_message,
    on_open=on_open
)
ws.run_forever()

This design, leveraging a stable endpoint like the one from AllTick, proves that architectural hygiene matters more than algorithmic complexity.

Production Hardening for Gold's Wild Nature

Gold’s high density of volatility requires a few more guards:

Signal Debouncing: A simple cooldown after a state change prevents rapid oscillation. This is a must-have hysteresis for any state machine.
Volatility Gate: Use an ATR filter. If the market is in a low-volatility chop, gate all signal output to prevent death by a thousand false signals.
Data Integrity Checks: Your state is only as good as your data feed. A half-second lag or a dropped tick during a high-impact news event will corrupt your entire state assessment. Monitoring your feed's health is not optional; it's fundamental.

Fix the data pipeline, and you’ll be amazed at how much smarter your simple strategies suddenly become.

Crypto Quant Systems: Polling vs WebSocket for Trading Pair Updates — and Why Hybrid Wins

Emily — Mon, 29 Jun 2026 04:57:24 +0000

The problem we don’t think about

When you’re building a digital asset quant system, you obsess over tick speed, order book depth, and backtesting accuracy. But there’s a quieter, sneakier issue that can undermine everything: your symbol list. As a data science lead for a crypto quant desk, I’ve seen multiple live incidents where a strategy fails with “symbol not found” because the local list of trading pairs was stale. This post digs into why trading pair updates are a non-trivial data engineering challenge and how we solved it.

The real cost of stale symbols

Crypto exchanges continuously adjust their instrument offerings. New coins get listed, dormant pairs get suspended, and sometimes pairs are delisted entirely. If you initialize your system once and never refresh, you create a growing divergence between your data view and reality. The damage is twofold: new pairs are invisible to your strategies, and invalid pairs generate noise that wastes compute and triggers false alerts. In a small setup with 20 symbols, you might never notice. Scale to 500 symbols across five exchanges, and it becomes a daily headache.

Inefficiency of naive approaches

Manual whitelisting breaks down immediately at scale. Timed polling every few hours reduces toil but introduces a latency window where the system is blind. I’ve seen new tokens rally 10% before our polling job caught up. And even when the job runs, you need logic to compare what’s changed — a raw list dump alone doesn’t tell you what’s new or removed.

Our technical solution: diff plus push

We resolved this by combining a lightweight diff engine with a push-based update stream.

The diff engine runs on every full list fetch:

# Existing cache
old_symbols = set(["BTCUSDT", "ETHUSDT"])
# Incoming list
new_symbols = set(["BTCUSDT", "ETHUSDT", "WIFUSDT", "PEPEUSDT"])

added = new_symbols - old_symbols
removed = old_symbols - new_symbols

print("New pairs:", added)
print("Removed pairs:", removed)

For the push component, we use WebSocket events. Data services like Alltick provide a WebSocket API that can deliver both tick-level market data and symbol change notifications through a single stream.

import websocket
import json

def on_message(ws, message):
    data = json.loads(message)
    if data.get("type") == "symbol_update":
        print("Symbol universe update:", data["symbols"])
    if data.get("type") == "tick":
        print(data["symbol"], data["price"])

ws = websocket.WebSocketApp(
    "wss://quote.alltick.co/stream",
    on_message=on_message
)
ws.run_forever()

This setup turned symbol updates from a background afterthought into a first-class event in our pipeline.

Changes in our workflow and architecture

We now keep symbol metadata in a dedicated service with a clear schema:

Field	Meaning
symbol	Pair identifier
status	Whether it’s tradable
update_time	Last status change
source	Origin of the data

This allows us to propagate status changes (e.g., active → suspended) to all dependent services. A lot of developers focus only on additions and removals, but status transitions are more dangerous — they can silently block orders while the symbol still looks valid in a cached list.

We’ve converged on a three-layer pattern: in-memory cache for low-latency lookups, periodic full sync to correct any drift, and WebSocket push for immediate awareness. It’s a hybrid that gives us both reliability and speed. The lesson? In digital asset quant engineering, the symbol list is not boring plumbing — it’s the foundation your strategies stand on. Invest in it accordingly.

You’re Probably Backtesting Forex with Too Short History — Here’s How We Verify

Emily — Fri, 26 Jun 2026 02:45:15 +0000

We’re a brokerage advisory team, and we spend a lot of time stress-testing forex strategies for our clients. If there’s one silent killer we’ve identified over the years, it’s this: forex API data history that’s too short.

Let’s walk through how we detect this problem and how we now structure our validation process.

What Our Clients Want

Traders who come to us want strategies that hold up in live conditions, not just in a perfect backtest. They need to know whether a model can survive a flash crash, a central bank surprise, or a prolonged low-volatility grind.

When a client shows us a strategy with stellar metrics, our immediate question is: How many years of data did you use? If the answer is two or three, we suspect the strategy hasn’t been stress-tested enough.

Where We Got Burned

We’ve been on both sides of the table. In our early days, we built strategies on APIs that provided only a few years of minute bars. The backtests were beautiful. When we later plugged in a decade of data, the performance imploded. That taught us the hard way: history length is not a detail, it’s a pillar of robustness. Short data windows give you the illusion of consistency by hiding the ugly parts.

The Invisible Differences Between APIs

Even when APIs claim “historical data,” the offerings differ in subtle ways:

Some provide data only from 2018, others from 2000;
Tick vs. K-line granularity mixes can distort entry/exit simulations;
High-volatility periods are often trimmed or smoothed;
The way the mid-price is calculated affects spread modeling.

These silent variations change your backtest distribution without ever throwing an error.

What Happens to Your Backtest

When we lengthen the history, we routinely observe:

The equity curve reshapes — smoothness turns into jaggedness;
Maximum drawdown is re-rated, often doubling;
Win rate adjusts downward;
Trade frequency and slippage models break.

If your strategy is short-term or high-frequency, insufficient history makes it memorize one specific micro-regime. Out of sample means out of luck.

Our Go-To Verification: Time Slicing

We now segment the historical data into windows: 1 year, 3 years, 5 years, and run the strategy on each. A strategy that only thrives in the 1-year window is considered regime-dependent.

To gather the raw ticks for this, we’ve used interfaces like AllTick API, which provide long tick histories via WebSocket. We store the data and then slice it. The core code snippet we use is:

import websocket
import json
import pandas as pd

data = []

def on_message(ws, message):
    msg = json.loads(message)
    data.append({
        "time": msg["ts"],
        "price": msg["price"],
        "volume": msg["volume"]
    })

ws = websocket.WebSocketApp(
    "wss://stream.alltick.co/ws",
    on_message=on_message
)
ws.run_forever()

df = pd.DataFrame(data)

# Slice by different time windows for backtesting
df_1y = df[df["time"] > "2026-06-01"]
df_3y = df[df["time"] > "2024-06-01"]

Seeing the performance divergence across these slices has saved us — and our clients — from deploying fragile strategies.

Our Advisory Upgrade

Our selection criteria for forex data sources have shifted from latency-first to depth-first. We care about how many market cycles are in the data, not just how recent it is. Short history can validate a strategy in a greenhouse, but only long history tests it in the wild.

If a strategy shines only in a narrow historical window, we tag it as a curve-fit artifact, not a real trading solution. That’s the standard we now hold for every strategy that carries a client’s capital.

Differentiating Auto-Matched and Odd-Lot Trades in Hong Kong Stock WebSocket Feeds

Emily — Thu, 18 Jun 2026 04:20:34 +0000

Working with real-time market data can feel like drinking from a firehose. When I started streaming Hong Kong equity trades over WebSockets, I quickly noticed that not all prints are equal. Some represent actual investor orders, while others are system-generated auto-matches or odd-lot transactions. In this post, I’ll share a straightforward approach to classify them on the fly.

The Problem
If you feed raw trades directly into a strategy or a volume indicator, odd lots and auto-matches will corrupt your metrics. For instance, a burst of auto-matches can inflate trade count without any real price movement, creating false breakouts. Manually filtering them is impossible at real-time speeds. So, automatic classification isn’t just nice to have — it’s essential.

Message Anatomy
Typical WebSocket trade data includes:

Field	Meaning
time	Trade time
price	Trade price
volume	Number of shares
trade_type	Often unreliable category
match_id	Matching identifier

Since trade_type rarely helps, I rely on three practical heuristics:

Volume check: HK stocks usually trade in board lots of 100 shares. Any trade with a non-round-lot volume (e.g., <100 shares) is tagged as an odd lot.
Time clustering: Auto-matched trades occur in dense bursts — multiple fills within milliseconds. Odd lots don’t show this pattern.
Counterparty inspection: If buyer and seller are both system accounts (like “SYS”), it’s an auto-match.

Implementation
I used the AllTick API to get a WebSocket connection for HK stocks. The Python snippet below subscribes to a symbol and tags every incoming trade:

from websocket import create_connection
import json

# Insert your AllTick API token here
API_TOKEN = 'your_api_token'
ws_url = f"wss://ws.alltick.co/stock?token={API_TOKEN}"

ws = create_connection(ws_url)

# Subscribe to real-time trades for HK stock 00700.HK
subscribe_msg = {
    "action": "subscribe",
    "symbol": "00700.HK",
    "type": "transaction"
}
ws.send(json.dumps(subscribe_msg))

def check_auto_match(tick):
    # Assume system auto-match counterparties are both "SYS"
    return tick.get('buyer') == 'SYS' and tick.get('seller') == 'SYS'

while True:
    data = ws.recv()
    tick = json.loads(data)

    volume = tick.get('volume', 0)

    if volume < 100:
        tick['tag'] = 'odd_lot'
    elif check_auto_match(tick):
        tick['tag'] = 'auto_match'
    else:
        tick['tag'] = 'normal'

    print(tick['time'], tick['price'], tick['volume'], tick['tag'])

Impact on My Work
Since adopting this classification layer, my downstream applications only consume “normal” trades, resulting in cleaner analytics and more trustworthy signals. The auto-match and odd-lot streams are still stored, allowing me to analyze market microstructure separately. It’s a simple yet powerful pattern that I recommend to anyone dealing with HK real-time data.

Handling UTC vs ET in Stock Market APIs: A Practical Guide with Python

Emily — Tue, 16 Jun 2026 06:13:36 +0000

If you’ve ever pulled historical stock data and found your charts shifted by an hour, you’re not alone. As a tech lead at a FinTech startup, I hit this wall hard. In this post, I’ll walk you through why it happens and how to fix it cleanly in your code.

Our Real-World Encounter

We were building a cross-market backtesting platform. Things were smooth until we combined two data feeds for US stocks. The minute bars were identical in price but off in time. This small discrepancy broke our intraday signal timing and caused a lot of confusion. The root cause: one API returned UTC, the other ET. Without explicit handling, our pandas dataframes treated both as naive timestamps.

The UTC/ET Dilemma Explained

UTC is consistent year-round, perfect for storage and computation.
ET reflects the actual US market hours but switches between EST (UTC-5) and EDT (UTC-4).

When you store data in ET, you risk duplicate or missing hours during DST transitions. When you store in UTC, you lose the quick visual mapping to Wall Street’s clock. Best practice: keep everything in UTC internally, convert to ET at the display or strategy layer.

How to Request and Process Time Zones

Always explicitly set the timezone parameter when fetching historical data:

GET /api/v1/history?symbol=AAPL&interval=1m&timezone=UTC

Once you have the data, use pandas to enforce UTC and derive an ET column:

import pandas as pd

df["timestamp"] = pd.to_datetime(df["timestamp"], utc=True)
df["timestamp_et"] = df["timestamp"].dt.tz_convert("US/Eastern")

For real-time streams, you want the provider to give you a reliable timestamp. AllTick’s WebSocket, for example, sends a ts field that’s ready to use:

from websocket import WebSocketApp
import json

def on_message(ws, message):
    data = json.loads(message)
    ts = data["ts"]
    price = data["price"]
    print(ts, price)

ws = WebSocketApp("wss://stream.alltick.co/v1/stock")
ws.run_forever()

Time Well Spent

After implementing this pattern, our data pipeline became much more robust. Adding a new data source no longer required guessing its timezone conventions—we simply normalized everything on ingestion. It’s a small fix with huge returns in terms of code maintainability and backtest accuracy. If you’re dealing with financial data, make timezone handling a first-class citizen.

Stop Polling Crypto Trades: Here’s How to Subscribe to Real-Time Trade Streams for Specific Pairs

Emily — Mon, 15 Jun 2026 07:23:51 +0000

As a university instructor in quantitative finance, I see students trip over the same data problem every term. They move from backtesting to live trading and suddenly their edge evaporates. Nine times out of ten, the root cause is their data ingestion method: they’re polling a REST API instead of receiving push-based trade streams. Let’s walk through the why and the how of WebSocket subscriptions for real-time trade data, focusing on subscribing only to the pairs you really need.

The Poll vs. Push Reality Check

I run a live comparison in class: a polling script requests BTCUSDT trades every 2 seconds, while a WebSocket client subscribes to the exact same pair. After a few minutes, the numbers speak loudly. The WebSocket client receives every single trade as it happens, with timestamps that are virtually real-time. The polling client skips multiple trades between requests and consistently lags. This isn’t a network issue — it’s the fundamental limitation of request-response cycles in a streaming world.

Don’t Drink from the Firehose: The Case for Targeted Subscriptions

Tempting as it may be to subscribe to “all pairs,” don’t. I’ve watched a student try this on a laptop: the script became unresponsive within seconds. The volume of tick-by-tick data for the entire market is massive and mostly irrelevant unless you’re market-making or running cross-sectional arbitrage. By subscribing only to the specific instruments your strategy involves, you dramatically lower CPU and memory usage, reduce noise in your signal processing, and make your system less fragile to reconnection bursts. Focus is a feature, not a limitation.

What’s Inside a Trade Push?

A typical WebSocket trade message is compact yet rich:

{
  "symbol": "BTCUSDT",
  "price": "30500.12",
  "quantity": "0.05",
  "side": "buy",
  "timestamp": 1686327890000
}

For developers building strategies, every field pulls its weight. symbol enables multiplexing multiple pairs over one connection. price and quantity are your immediate inputs for computations. side is a direct indicator of aggressive direction, crucial for order flow imbalance signals. timestamp allows latency monitoring and correct temporal sequencing. You don’t need to join multiple data sources; this single message gives you an actionable atomic event.

Scaling to Multiple Pairs and Surviving Disconnects

To handle multiple symbols, send a subscription request with a list of pairs. Messages come back tagged with symbol, which you can use to dispatch to per-pair logic. At high message rates, process data asynchronously: let your WebSocket callback put messages into a queue, and let a pool of workers drain it. For disconnects, implement a reconnect strategy in on_close or on_error. Once reconnected, resubscribe and filter out duplicate trades using a sliding window of recent timestamps. This pattern keeps your pipeline robust without complex state management.

Run This Minimal Subscriber Now

The following snippet uses the AllTick WebSocket API — a clean, developer-friendly endpoint I often use in workshops:

import websocket
import json

def on_message(ws, message):
    data = json.loads(message)
    print(f"Trade: {data['symbol']} {data['price']} qty {data['quantity']}")

def on_open(ws):
    subscribe = {
        "action": "subscribe",
        "symbol": "BTCUSDT",
        "type": "trade",
        "id": 1
    }
    ws.send(json.dumps(subscribe))

ws = websocket.WebSocketApp("wss://api.alltick.co/ws",
                            on_message=on_message,
                            on_open=on_open)
ws.run_forever()

Save it, install websocket-client, and run. The moment you see live trades streaming in, you’ll understand the difference push-based data makes. Check out AllTick’s docs to explore more pair codes and advanced subscription controls. Your strategies deserve data that arrives when the market moves, not seconds later.

Stop Letting Halted Stocks Pollute Your Real-Time Feed

Emily — Thu, 11 Jun 2026 03:48:29 +0000

You’ve wired up a shiny WebSocket stream, ticks are flying in, and then you notice — some symbols keep sending the same price with zero volume. If you’re building a trading app or a quant system, you need to know exactly how to handle these halted symbols without bloating your pipeline. Let me share a practical, demand-driven approach that works across multiple APIs.

The Real Demand: Tradable vs. Non-Tradable

Every downstream consumer — be it a UI component, a backtest engine, or a risk monitor — expects a clean separation. If a halted stock slips through, your calculations get contaminated and your users see bogus opportunities. So the spec is clear: detect suspension as early as possible and quarantine the data.

Where It Hurts

I’ve seen teams try to infer halts using only price and volume deltas. The problem? Low-volume stocks can mimic a halt, and certain APIs keep streaming prices for days after a suspension. This leads to “zombie” stocks in the order book, wasted CPU cycles, and inflated bandwidth bills. Without a dedicated field, you’re playing a guessing game.

Data Patterns to Watch For

Through countless integrations, I’ve mapped the typical halt behaviors:

Behavior	Meaning	Use It?
Static last price	No trade update	Not reliable alone
Cumulative volume frozen	No new trades	Good supplementary check
`suspend` or `halt` field	Official suspension signal	Primary indicator
Timestamp-only updates	Heartbeat with no data	Ignore for logic

Rule of thumb: if a status field exists, trust it. Otherwise, combine volume flatline with a price staleness timer.

Upgrading Your Pipeline

I now design my ingestion pipelines with three distinct layers:

Ingestion: On each tick, immediately inspect suspend (or derived halt flag) and tag the message.
Business logic: Filter all calculations — factor generation, spread analysis, basket NAV — using halted == false.
Connection management: Unsubscribe from halted symbols on the fly. If your API provides a halt field natively (for example, AllTick’s payload includes suspend), you can act on it in a single callback without any complex state tracking.

import websocket
import json

def on_message(ws, message):
    data = json.loads(message)
    for tick in data.get("ticks", []):
        if tick.get("suspend"):  # Check if suspended
            print(f"{tick['symbol']} halted")
        else:
            print(f"{tick['symbol']} last price: {tick['last_price']}")

ws = websocket.WebSocketApp("wss://api.alltick.co/stock", on_message=on_message)
ws.run_forever()

Adapting Across Providers

Not all APIs are created equal. Some stop sending data entirely upon halt, others mutate a status code, and a few keep pushing everything except volume. My go-to strategy is to run a benchmark capture in a staging environment, cluster symbols into active, halted, and special (new/delisted), and apply targeted policies per cluster. Treat halt detection as a core data-quality module, and your whole stack will thank you.

How to Reconstruct a Crypto Order Book in Python Using WebSockets and SortedDict

Emily — Mon, 08 Jun 2026 03:17:33 +0000

Hey devs! 👋 We want to share a practical recipe that has become a staple in our quantitative data pipeline: streaming a cryptocurrency order book in real time and maintaining an exact local copy using Python. If you’ve ever needed reliable market depth for your app, bot, or analysis notebook, this setup will give you full ownership of the data.

The Problem with Polling

REST endpoints give you a static snapshot of the order book. That’s okay for low-frequency dashboards, but if you need up-to-the-millisecond accuracy—for example, when computing slippage or order flow signals—polling introduces gaps that ruin the picture. You need a persistent WebSocket stream that pushes every change as it happens.

Ingredients

WebSocket client to receive live data.
Snapshot for initializing the book.
Diffs for keeping it current.
SortedDict to store bids and asks efficiently.

We used sortedcontainers because it keeps items in order and supports fast insertions/deletions. Bids are sorted descending, asks ascending.

from sortedcontainers import SortedDict

# Bids: highest price first
bids = SortedDict(lambda x: -x)
# Asks: lowest price first
asks = SortedDict()

Step-by-Step Code

Connect to a WebSocket endpoint that provides snapshot and diff messages. We’ve had good experiences with AllTick’s feed, as the data format is clean and well-documented. Here’s the full minimal example:

import websocket
import json
from sortedcontainers import SortedDict

bids = SortedDict(lambda x: -x)
asks = SortedDict()

def on_message(ws, message):
    data = json.loads(message)
    # Update both sides of the book
    process_orderbook(data)

def process_orderbook(data):
    global bids, asks
    for update in data.get("bids", []):
        price, size = update
        if size == 0:
            bids.pop(price, None)
        else:
            bids[price] = size
    for update in data.get("asks", []):
        price, size = update
        if size == 0:
            asks.pop(price, None)
        else:
            asks[price] = size

ws = websocket.WebSocketApp("wss://api.alltick.co/crypto/orderbook",
                            on_message=on_message)
ws.run_forever()

Making It Production-Ready

The snippet above works for experimentation, but for anything serious you’ll want:

Sequence verification on diffs to detect gaps.
Automatic snapshot reload if a gap is found.
Heartbeat and auto-reconnect for the WebSocket connection.

Implementing these guards turns a fragile script into a self-healing service.

Visual Feedback

During development, we often plot the order book using matplotlib to sanity-check the data:

import matplotlib.pyplot as plt

def plot_orderbook():
    plt.plot(list(bids.keys()), list(bids.values()), color='green', label='Bids')
    plt.plot(list(asks.keys()), list(asks.values()), color='red', label='Asks')
    plt.legend()
    plt.show()

Green shows the bid side, red the ask side. Any sudden gap or spike immediately reveals a synchronization issue.

Wrapping Up

With a few dozen lines of Python, you can maintain a real-time, exchange-accurate order book locally. This opens the door to custom indicators, low-latency alerts, and strategy backtesting on consistent data. Give it a try—it’s one of those building blocks that pays dividends across many projects.

Handling Time Zone Differences in Forex APIs: A Practical Developer’s Guide

Emily — Thu, 04 Jun 2026 04:20:28 +0000

Handling Time Zone Differences in Forex APIs: A Practical Developer’s Guide

When I started building a multi-source forex data pipeline for a brokerage team, I kept running into a subtle but destructive bug: the same EUR/USD tick could appear with timestamps hours apart depending on which provider it came from. The strategy would see a quote that supposedly happened at 15:30, but another source recorded the identical event at 07:30. Without a proper time alignment layer, the whole backtest engine was working with a warped timeline.

If you’re ingesting forex quotes from different APIs, this guide will show you how I solved it—from raw input to storage—with a focus on practical, reusable patterns.

Step 1: Know Your Timestamp

Forex APIs love to play format roulette. You might get a second-based Unix epoch, a millisecond epoch, a UTC ISO string (with Z), or a local time string that omits the zone entirely. I’ve learned to read the docs carefully and, more importantly, to never trust an unqualified local time. My approach is to normalize every incoming time to a UTC timestamp in milliseconds at the earliest entry point. That single act eliminates a universe of off-by-hours errors.

Step 2: Convert Time Zones Safely

Python’s pytz gives you the tools to shift between zones without stress. Parse the string, pin it to UTC, then convert. The snippet below takes a UTC quote and transforms it to Shanghai time, which is often needed when aligning with Asian trading sessions:

from datetime import datetime
import pytz

utc_time = datetime.strptime("2026-06-03T07:30:00Z", "%Y-%m-%dT%H:%M:%SZ")
utc_time = utc_time.replace(tzinfo=pytz.UTC)

# Convert to UTC+8 for local analysis
local_time = utc_time.astimezone(pytz.timezone("Asia/Shanghai"))
print(local_time)

One pro tip: if the API sends both a timestamp and a formatted string, choose one as the authoritative source. Mixing the two can lead to micro-drift that is extremely tricky to catch.

Step 3: Align Multiple Price Feeds

When you have quotes arriving from London and New York simultaneously, their timestamps won’t match perfectly. I take all records, bring them to UTC, sort them, and then fuse them with an alignment strategy. The three I commonly use are:

Nearest valid: quickest for real-time; the strategy receives the closest available price.
Mean fill: averages quotes within a small window—excellent for backtesting.
Linear interpolation: bridges gaps for continuous-time models.

Choosing the right strategy depends on your latency tolerance and modeling needs. I tend to use mean fill in research and nearest valid in live trading.

Step 4: Store Data with a Time-First Mindset

I set the UTC timestamp as the primary key in my quotes database. PostgreSQL’s timestamptz is perfect for this—it keeps everything consistent even when you later add new data sources. When the front-end or reporting layer needs local time, a simple query-time conversion does the job. For high-volume tick storage, I limit columns to time, price, and volume, and I partition tables by date to keep performance predictable.

Step 5: Test It with a Real-Time Forex Stream

To validate the whole setup, I connected to AllTick’s WebSocket feed. It pushes UTC millisecond timestamps natively, so the integration was smooth. Here’s the handler I used:

import websocket
import json
from datetime import datetime
import pytz

def on_message(ws, message):
    data = json.loads(message)
    utc_ts = data['timestamp']  # milliseconds
    utc_time = datetime.utcfromtimestamp(utc_ts / 1000).replace(tzinfo=pytz.UTC)
    local_time = utc_time.astimezone(pytz.timezone("Asia/Shanghai"))
    print(local_time, data['symbol'], data['price'])

ws = websocket.WebSocketApp("wss://api.alltick.co/realtime", on_message=on_message)
ws.run_forever()

This ensures every tick is aligned from the moment it arrives, giving both the strategy engine and the monitoring tools a shared, consistent timeline.

Wrapping Up

Forex time alignment isn’t glamorous, but it’s the bedrock of reliable quant workflows. Adopting UTC as your internal standard, enforcing a single time representation, and defining explicit alignment rules for multi-source data will save you from countless hours of debugging strange signal behavior. Combine that with mindful API latency monitoring, and your data foundation will be solid enough to build real alpha on.

How I Subscribed to Multiple US Stocks Ticks with a Single WebSocket Connection

Emily — Mon, 01 Jun 2026 06:48:40 +0000

My daily routine as a retail algo trader used to be haunted by latency. I’d watch my logs fill up with stale prices while the market moved on. The culprit? HTTP polling. I was firing hundreds of requests per minute just to keep up with 10-15 symbols, and still the data had gaps. After collecting stats, I realized the average delay was over 700ms — a dealbreaker for tick-based strategies. So I ditched polling and embraced WebSocket streaming with the AllTick market data API. This post shows you exactly how I did it.

What’s Wrong with Polling?

When you poll, you’re taking snapshots at fixed intervals. Between those snapshots, anything can happen. High-frequency updates demand a continuous stream, not discrete photos. Polling also scales poorly: more stocks mean more loops, more sockets, more waste. WebSocket solves this with a single long-lived connection that pushes updates as they occur, preserving timeline integrity.

Code Walkthrough: One Connection to Rule Them All

Here’s my Python client. It subscribes to multiple US stocks at once and prints each tick. Simple, right?

import websocket
import json

def on_message(ws, message):
    # Parse tick update
    data = json.loads(message)
    print(data["symbol"], data["price"], data["time"])

def on_open(ws):
    # List of symbols to track
    symbols = ["AAPL", "TSLA", "AMZN"]

    req = {
        "action": "subscribe",
        "symbols": symbols
    }

    ws.send(json.dumps(req))

ws = websocket.WebSocketApp(
    "wss://api.alltick.co/ws/stock",
    on_open=on_open,
    on_message=on_message
)

ws.run_forever()

Once running, you’ll see a continuous log of price updates — no more gaps or bursts of stale data.

Keep It Running in Production

For serious use, I added a few enhancements:

Auto-reconnect: Network drops? No problem. Reconnect in on_close with a delay.
Queue buffer: When ticks flood in, push them to a queue and let a worker thread handle processing.
Dedup logic: Avoid processing the same tick twice by caching recent (symbol, timestamp) pairs.
Multiple connections: For many symbols, split them among several connections to balance the load.

Where the Ticks Go

I pipe incoming ticks into two pipelines: one stores them in a TSDB for backtesting, the other drives my live trading engine. A small Redis buffer prevents database write spikes from slowing down my strategy execution.

One important tip: always normalize timestamps to a common format (I use UTC microseconds) right after receipt, so all downstream services speak the same language.

From Polling to Pushing

Switching to WebSocket turned my trading system from a laggy, unreliable mess into a smooth, real-time machine. Slippage is down, confidence is up, and I spend less time debugging data feeds. If you’re building any tick-sensitive application, take the leap to WebSocket — you won’t look back.