Intro: The Hidden Time-Series Bias That Skews Institutional Backtests
As quantitative researchers and backend engineers working within asset management teams, our day-to-day work centers on building robust backtesting frameworks and real-time market monitoring pipelines for US equities. We’ve run into a recurring, hard-to-trace bug across multiple internal projects: identical trading algorithms deliver wildly inconsistent performance metrics when fed data from different market sources.
After rounds of comparative backtesting and log auditing, we ruled out flawed candlestick aggregation functions as the root issue. Instead, the discrepancy stems from inconsistent handling rules for pre-market, regular session, and after-hours tick data. Most junior engineers overlook session boundary standardization, which quietly warps price distributions and undermines the credibility of all downstream strategy analysis.
The Structural Differences Between US Stock Trading Sessions
Unlike most single-session equity markets, US equities trade across three distinct daily windows, each with unique liquidity profiles and sensitivity to market news events.
Pre-market: 04:00–09:30 ET, thin order book depth; overnight news and preliminary earnings often trigger sharp, unbalanced price moves
Regular trading hours: 09:30–16:00 ET, peak institutional liquidity; all classic technical indicators are calibrated against this window
After-hours: 16:00–20:00 ET, dominated by post-market earnings releases, prone to rapid short-term volatility
Nearly all modern market data APIs return full tick records covering all three windows, yet many pipelines filter out extended-hour data by default. For example, a stock could rally from $100 to $103 during pre-market. If your code discards those ticks, the opening candle of the regular session will still start near $100. This barely impacts casual chart viewing, but it fundamentally distorts datasets used for factor research, indicator calibration, and multi-year backtesting.
Standard Timestamp Normalization: The Core ETL Foundation
Before aggregating any ticks into candlesticks, we enforce universal timestamp normalization across both offline historical datasets and live streaming feeds. There is no universal aggregation formula for extended-hour data—processing logic must be customized for your business use case.
Our team follows a fixed, reusable ETL pipeline for all US equity tick processing:
Consume raw tick payloads from market APIs
Parse raw millisecond Unix timestamps
Convert timestamps to Eastern Time zone
Tag each tick with its corresponding trading session
Aggregate tagged records into OHLC candlesticks on demand
Time zone alignment is the most error-prone step here. US market rules are governed by America/New_York time, but cloud servers store timestamps in UTC to avoid regional bias. If you render candlesticks directly using raw server timestamps, daylight saving time shifts will introduce permanent 1-hour offsets. Our production standard: persist all raw ticks as UTC timestamps; convert to ET only during session tagging and candlestick rendering.
Custom Candlestick Aggregation Rules Per Workflow
Blindly mixing low-liquidity extended-hour ticks with high-volume regular session prints pollutes volume-based indicators like volume moving averages and turnover ratios. We maintain four standardized processing templates for different research workflows:
Long-term multi-year trend analysis
Only aggregate regular-hours ticks into candlesticks. Pre/after-hour data is archived in separate tables and excluded from indicator calculations, matching the design intent of traditional technical analysis tools.
Intraday & high-frequency strategy backtesting
Include every tick across pre, regular, and after-hours sessions to reconstruct the full daily price trajectory, fully capturing overnight gap risk.
Real-time live market dashboards
Retain unfiltered raw tick streams with no early aggregation to support sub-second visualization.
Earnings event research
Isolate after-hours tick data into independent datasets to avoid signal contamination from intraday trading activity.
Live Tick Stream Integration Using
To eliminate mismatched logic between offline historical data and real-time feeds, we reuse the exact timezone and session tagging functions across both pipelines.
For live US equity tick ingestion, our production stack leverages AllTick API’s persistent WebSocket subscription endpoint to maintain consistent data standards end-to-end.
Below is a minimal working Python snippet for real-time tick subscription and timestamp conversion:
import websocket
import json
from datetime import datetime
import pytz
UTC_ZONE = pytz.utc
NY_ZONE = pytz.timezone("America/New_York")
def tick_receive(ws, raw_data):
data = json.loads(raw_data)
symbol = data.get("symbol")
price = float(data.get("price"))
ts_ms = data.get("timestamp")
utc_dt = datetime.fromtimestamp(ts_ms / 1000, tz=UTC_ZONE)
ny_dt = utc_dt.astimezone(NY_ZONE)
print(f"Ticker:{symbol} Price:{price} NY Time:{ny_dt}")
if __name__ == "__main__":
ws_client = websocket.WebSocketApp(
"wss://api.alltick.co/stock/websocket",
on_message=tick_receive
)
ws_client.run_forever()
Frequently Overlooked Edge Cases That Break Candlestick Integrity
After deploying dozens of institutional-grade backtesting pipelines, we’ve documented three common hidden pitfalls:
Cross-midnight trading day labeling
Most APIs return timestamps in UTC, while US trading days are defined by Eastern Time. Ticks crossing UTC midnight need date correction to avoid misassigned daily candlestick blocks.
Extended-hour data API parameters
Many market endpoints only return regular session ticks by default. You must explicitly enable extended-hour request flags to retrieve pre/after prints, otherwise gap volatility data will be permanently missing.
Streaming fault tolerance
Production WebSocket pipelines must implement auto-reconnection, duplicate tick deduplication, and timestamp sorting. Unsorted or duplicate ticks generate malformed, unreliable candlestick charts.
Closing Takeaways For Quant & Data Engineers
From an institutional quantitative development perspective, market data APIs are far more than simple price fetching tools. Their true value hinges on your ability to correctly interpret session-based time logic baked into US equity market rules.
There is no one-size-fits-all method to merge pre-market and after-hours ticks. Always design aggregation logic aligned with your research or trading objectives. We recommend finalizing trading session tagging and timestamp normalization rules before writing any candlestick generation code. Standardized time-series preprocessing creates reproducible indicator outputs and reduces performance drift between backtest simulations and live market execution.
Top comments (0)