As quants and algo developers, we’ve all hit the wall where our strategy’s optimal lookback window doesn’t match any of the standard K-line periods provided by an exchange or data vendor. Whether it’s 30 seconds, 3 minutes, or 42 minutes, the requirement is crystal clear: give me bars of exactly this duration. In our fund’s R&D team, we transitioned completely away from third-party candles and now synthesize every single bar ourselves from the raw tick stream. In this post, I’ll walk you through the rationale, the mathematical mapping, and the Python code you need to replicate this approach in your own stack. The implementation is simpler than you might think, and it unlocks a new level of flexibility for your strategies.
Client Requirement: Why standard periods hold strategies back
Our internal “clients” are portfolio managers and systematic traders who increasingly request non-standard bar intervals. A stat-arb strategy might need 37-second candles to align with a specific exchange’s auction frequency; a momentum model may discover its Sharpe peaks at 104 minutes; an execution algorithm wants 10-second bars to calculate real-time VWAP slippage. When you rely on pre-baked OHLCV endpoints, you’re constrained to a discrete, sparse set of periods. Worse, you’re forced to use a data generation pipeline that may differ between historical and live environments. We needed a single, unified way to produce any candle period — on demand and with identical behavior in both backtests and real-time.
Practitioner Pain Points: What’s wrong with pre-aggregated K-lines?
There are three major pitfalls that pushed us toward self-aggregation:
- Inconsistent window boundaries. Different data sources define the start of a “5-minute” bin differently, especially across exchanges with distinct matching-engine timestamps. Those micro-offsets corrupt cross-market signals.
- Loss of intra-bar dynamics. A candle’s high/low/open/close tells you nothing about the path price took inside the window. For short-term mean reversion or order flow strategies, that path is everything. Pre-aggregated data discards it permanently.
- Backtest/live divergence. Historical K-lines are batch-produced; live candles stream continuously. If those two code paths aren’t identical, your backtest results become unreliable. We’ve seen models that looked incredible in simulation fail outright in production simply because the bar-opening rule differed by a single tick.
Data Foundation: The tick-to-candle mapping
Tick data carries three essential fields: price, volume, and timestamp. A candle is a windowed reduction of those fields. The exact arithmetic is:
| Field | Calculation |
|---|---|
| Open | First trade price inside the window |
| High | Maximum trade price inside the window |
| Low | Minimum trade price inside the window |
| Close | Last trade price inside the window |
| Volume | Sum of all traded quantities in the window |
For instance, to build a 5-minute bar starting at 10:00, you’d collect all ticks with timestamps in [10:00:00.000, 10:04:59.999] and apply those five formulas. This simple mapping is the entire secret sauce.
Service Upgrade: Implementing the aggregator in Python
We built our aggregator as a reusable Python module that works identically for historical files and live websocket feeds.
Step 1: Normalize timestamps and create time buckets
All timestamps are first converted to UTC milliseconds. For a target period (say 60 seconds), we bucket ticks by applying:
period = 60
bar_time = timestamp - (timestamp % period)
Step 2: Aggregate ticks into candles
Using defaultdict, we group ticks by their bucket key and then compute the OHLCV statistics:
from collections import defaultdict
ticks = [
{"time":1710000001,"price":100,"volume":2},
{"time":1710000010,"price":102,"volume":3},
{"time":1710000030,"price":101,"volume":1}
]
period = 60
bars = defaultdict(list)
for tick in ticks:
key = tick["time"] - (tick["time"] % period)
bars[key].append(tick)
for timestamp, data in bars.items():
prices = [item["price"] for item in data]
volumes = [item["volume"] for item in data]
print({
"open": prices[0],
"high": max(prices),
"low": min(prices),
"close": prices[-1],
"volume": sum(volumes)
})
For large-scale historical processing, we swap the in-memory dictionary for a generator that streams ticks from disk and writes completed candles directly to a time-series database. The mathematical operation remains the same.
Step 3: Real-time streaming via WebSocket
In live trading, we open a persistent WebSocket connection to receive a continuous flow of ticks. We use the AllTick low-latency WebSocket feed to get real-time trade data, which then feeds directly into our aggregator:
import websocket
url = "wss://quote.alltick.co/socket.io"
ws = websocket.create_connection(url)
ws.send('{"cmd":"subscribe","symbol":"BTCUSDT"}')
while True:
data = ws.recv()
print(data)
On each tick, we determine its window for every watched interval. If the tick belongs to the currently active candle, we update the running high, low, and volume in memory. Once a tick’s timestamp exceeds the window boundary, we “seal” the candle, publish it, and start a new one. This state machine runs inside the data ingestion process to avoid network hops and ensure microsecond-level latency.
Watch out for these common pitfalls:
- Empty windows: In backtesting, we insert flat candles for periods with zero trades to preserve time-series continuity. Live systems can skip them if the strategy is tolerant.
- Volume semantics: Always confirm whether tick volume is incremental or cumulative. We reconcile daily sums against exchange-reported volumes on every new data source.
- Timestamp precision: Mixing second and millisecond timestamps will shift your bars subtly but devastatingly. Normalize to a single precision upfront.
Wrapping Up
Owning the tick-to-candle pipeline has made our strategies more agile and our research more trustworthy. You can test any interval, confident that the same code will run in production. If you’re building trading systems, I highly recommend investing this small piece of infrastructure. The code above is a minimal starting point — extend it, wrap it in your favorite streaming framework, and never be limited by standard bar sizes again.
Top comments (0)