When you’re building backtesting infrastructure, there’s a special kind of frustration that comes from staring at a perfectly flat equity curve and realizing the strategy didn’t fail — the data pipeline did. We ran into this exact situation when working on a US equity quantitative system. Historical bars were missing for no obvious reason, the API didn’t error out, and our results were silently corrupted.
In this post, we’ll walk through how we uncovered these hidden gaps, the technical root causes we discovered, and the engineering patterns we now use to ensure time-series continuity. This guide is written from our perspective as quant engineers who have spent too many nights debugging data that should have been complete.
Scenario: The Backtest That Was Too Smooth
We were testing a momentum-based strategy on US stocks. One segment of the backtest showed a near-zero return stretch that didn’t match market conditions. Upon inspection, the historical minute data we’d pulled via a standard REST API had a multi-day gap. No exceptions, no HTTP errors — just a silent skip. We needed to understand why and make sure it never happened again.
Requirements: Continuous Historical Sequences
For rigorous quantitative research, every timestamp matters. A missing hour can bias Sharpe ratios, factor returns, and walk-forward optimization. Our pipeline must guarantee that the time axis is unbroken from the first bar to the last. This requirement applies regardless of the API provider or asset class.
Data Pain Points: The Anatomy of Gaps
After dissecting many failures, we categorized the root causes into four buckets.
1. Source-Level Incompleteness
Many vendors do not provide full tick coverage for all session segments. Pre-market, after-hours, and low-volume periods may be delivered as sparse summaries or skipped entirely. The timeline naturally contains blanks that cannot be recovered downstream.
2. Pagination and Cursor Errors
Historical endpoints use pagination tokens or time cursors. A single failed request combined with an incorrect retry that advances the cursor will permanently skip a data segment. Because the HTTP response code eventually becomes 200 for the next page, monitoring tools see no problem.
3. Timezone Misalignment
US equities operate on Eastern Time, while servers frequently work in UTC. If request boundaries aren’t normalized and DST isn’t accounted for, intervals shift. Overlaps or gaps appear in what should be a seamless sequence. These are “false gaps” — the data exists but is misaligned.
4. Aggressive Server-Side Filtering
Some APIs clean the data before serving it, removing price spikes or duplicate trades. When the filtering threshold is too aggressive, especially in illiquid stocks, legitimate records disappear, creating visible holes.
Solutions: Building a Gap-Resistant Pipeline
We implemented a three-layered approach to detect and prevent these issues.
Layer 1: Universal Time Normalization
All timestamps are converted to UTC at the ingestion boundary, with original timezone metadata preserved. Request windows are expressed strictly in UTC, and market session calendars handle DST shifts explicitly. This eliminates the majority of false gaps.
Layer 2: Pagination Continuity Validation
Every time we receive a page of historical data, the client compares its first timestamp with the previous page’s last timestamp. If the gap exceeds the expected sampling interval, the page is flagged and re-fetched. This turns silent cursor slips into actionable alerts.
Layer 3: Live Tick Stream Cross-Reference
We maintain a parallel, unfiltered tick-level feed to serve as a ground-truth timeline. For example, by subscribing to AllTick’s WebSocket, we receive every trade in real time and cache it locally. We then compare the historical data against this live reference. Any period where the historical feed is empty but the live feed shows activity is immediately identified as a data gap.
Implementation of the live tick listener:
import websocket
import json
def on_message(ws, message):
data = json.loads(message)
timestamp = data.get("timestamp")
price = data.get("lastPrice")
print(timestamp, price)
ws = websocket.WebSocketApp(
"wss://api.alltick.co/stock",
on_message=on_message
)
ws.run_forever()
Bonus: Stitching Historical and Real-Time Streams
Another common pitfall is the transition between historical and live feeds. If the historical endpoint stops at time T and the real-time feed starts at T+delta, an artificial gap emerges. We enforce overlapping windows and boundary alignment to keep the overall sequence welded tight.
Key Takeaways
- Don’t trust 200 OK alone: Validate time continuity explicitly.
- UTC everywhere: Normalize all time boundaries.
- Automate pagination checks: A simple timestamp comparator saves days of debugging.
- Use a live tick baseline: Real-time data is your best auditor for historical completeness.
In the end, reliable quantitative research depends less on the sophistication of your models and more on the integrity of the time chain. Fix the gaps, and the alphas become much clearer.
Top comments (0)