When your execution algorithm relies on precise historical patterns, nothing stings more than discovering your backtest was trained on incomplete data. In our team, we’ve encountered numerous cases where a seemingly profitable strategy collapsed the moment we fed it a complete K-line series. In this guide, I’ll show you exactly how we detect and fix candlestick gaps in crypto data—without ever injecting synthetic prices.
Why Gaps Appear in a 24/7 Market
A 1-minute K-line should arrive every 60 seconds. In reality, gaps creep in through WebSocket disconnections, API rate limits, logger crashes, or timezone mismatches. The gap doesn’t mean the exchange paused; it means your recording pipeline missed the snapshot. If your execution logic trusts these gaps, it will make decisions on a distorted version of reality.
The Manual Grind We Left Behind
Early on, we checked for gaps by exporting data and visually scanning timestamps. This was slow, inconsistent, and soul-crushing. As execution engineers, our time should be spent optimizing latency and slicing algorithms, not playing data detective. The inefficiency was silently eroding our ability to ship robust strategies.
Our Automated Repair Workflow
We built a validation layer that automatically audits every historical dataset before it enters a backtest. The mandatory checks are summarized below:
| Check | Goal |
|---|---|
| Timestamp interval consistency | Detect any gap by comparing actual diff to expected period |
| Duplicate timestamps | Avoid double-counting bars |
| OHLC price logic (High ≥ Low, etc.) | Reject corrupted rows |
| Volume behavior sanity | Confirm trades occurred where prices moved |
If a gap is found, we skip interpolation entirely. Instead, we reconstruct the missing candle from tick data. The principle is straightforward: a candlestick is an aggregation of individual trades within a window. We collect all ticks in the missing interval and derive open (first trade), close (last trade), high, low, and summed volume. To have ticks always available, we maintain a continuous WebSocket feed.
import websocket
import json
url = "wss://quote.alltick.co/quote-stock-b-ws-api"
def on_message(ws, message):
data = json.loads(message)
print(data)
ws = websocket.WebSocketApp(
url,
on_message=on_message
)
ws.run_forever()
A provider like AllTick delivers the raw tick stream, which we persist with all timestamps normalized to UTC. This timestamp unification is critical—many apparent gaps are really timezone illusions, and converting everything to UTC instantly dissolves them.
How This Changed Our Daily Work
With the automated pipeline in place, we’ve completely eliminated manual data inspection. Our backtests now reflect genuine market conditions, and our execution algorithms behave identically from simulation to production. We’ve found that investing in data continuity often yields greater performance improvement than months of parameter optimization. If you’re building a trading bot, treat your data pipeline as the product’s foundation—it makes everything else stand.
Top comments (0)