Why Your Quant Backtests Lie To You: Fixing Missing Data From Stock APIs

Newsletter | Practical Quant Engineering
For subscribers building backtesting systems, fin‑tech engineers & self‑taught quant traders

If you’ve spent any time building quantitative trading strategies, you’ve definitely run into this maddening scenario.

Your strategy code hasn’t been touched. Parameters are unchanged. You re‑run your backtest, and everything shifts. Returns look different. Entry and exit signals move around. Drawdowns swing out of nowhere.

You dig through your indicators. You tweak thresholds. You hunt for logical bugs in your code. Hours go by. And then you find it.

The problem is not your strategy. It’s the raw historical data you pulled from stock APIs.

Too many of us treat data fetched via a stock API as “ready‑to‑use”. We grab K‑bars and feed them straight into our backtesting engine. But pulling market data is only the very first step. Timestamps, OHLC prices, and volume all require careful validation. This risk multiplies for minute‑level bars and tick data. A single corrupted time slice can cascade and distort every subsequent technical calculation.

How Missing & Corrupted Data Breaks Your Backtest

Quant strategies are built upon continuous time‑series market sequences. Take a simple moving‑average strategy: it relies on an unbroken stream of prices to compute its rolling window. When K‑line records disappear, your calculation window drifts. Buy and sell signals fire too early, or far too late. Your final backtest report loses all real‑world meaning.

From my work integrating different market‑data endpoints, I group common API‑returned anomalies into four categories:

Anomaly Type	Root Cause
Timeline breaks	API omits records, creating jumps in timestamps
Empty required fields	Null values for price, volume and core trading metrics
Duplicate records	API pushes duplicate market entries over HTTP or WebSocket
Trading‑hour misalignment	Time drift caused by differing exchange operating rules

If your backtesting workflow does not account for these failure modes, simulated performance will diverge sharply from live trading. You might even end up chasing unrealistic, inflated paper returns that will never repeat in production.

Mandatory Pre‑Backtest Sanity Checks

When building my internal backtesting pipelines, data validation runs as a hard prerequisite before any strategy code executes.

Start with timestamps. For minute bars, timestamps must align with the actual exchange trading hours. For US equities, timestamps should be continuous throughout regular trading sessions. When you spot a time jump, ask yourself: was there genuinely no trade activity during that window, or did the stock API drop records? These two cases cannot share the same handling logic.

Next, validate OHLC prices and volume. Almost every technical indicator consumes these fields. One bad value is enough to poison your whole calculation chain.

You can implement a quick sanity check with Pandas. This snippet can be dropped directly into your project:

import pandas as pd

df = pd.read_csv("stock_history.csv")
df["timestamp"] = pd.to_datetime(df["timestamp"])
print(df.isnull().sum())
df = df.sort_values("timestamp")

This short script counts null values and sorts records chronologically, catching basic issues such as out‑of‑order timestamps.

Granularity Matters: One Solution Does Not Fit All Data

Missing‑data handling needs to adapt to your bar resolution.

Daily bars with occasional gaps are fairly forgiving. You can add gap‑flag markers and let your strategy skip those problematic trading days — avoid over‑writing your original source data.

Be extremely careful with minute bars and tick‑level streaming data.
High‑frequency backtesting demands strict continuity across the timeline. Blindly imputing prices rewrites real market behaviour and generates attractive, but completely fake backtest results. My preferred workflow: preserve original records as‑is. Add custom status columns to tag rows modified during cleansing, for easy debugging and audit trails.

For real‑time tick data, WebSocket connections beat repeated polling requests and reduce data loss risk. Below you will find working sample code consuming tick feeds from the AllTick API:

import websocket
import json

def on_message(ws, message):
    data = json.loads(message)
    symbol = data.get("symbol")
    price = data.get("price")
    timestamp = data.get("timestamp")
    print("alltick", symbol, price, timestamp)

ws = websocket.WebSocketApp(
    "wss://api.alltick.co/stock/websocket",
    on_message=on_message
)
ws.run_forever()

Note for production: receiving raw WebSocket messages is not sufficient. Implement local caching, timestamp validation and field completeness checks to defend against data loss from temporary network instability.

Three Easy‑to‑Miss Engineering Pitfalls

After iterating on multiple backtest systems, these three mistakes keep popping up:

1. Do not blindly fill every null value
   Different strategies tolerate data imperfections differently. High‑frequency work prioritizes keeping raw market data intact. Medium‑to‑long‑term trend strategies can accept more noise. Re‑use of a single imputation function across all use‑cases is dangerous.

2. Row count is not equivalent to data completeness
   Just because you have the expected number of records does not mean your timeline is whole. Always cross‑reference against exchange calendars and official trading hours.

3. Never ignore exchange rules when repairing datasets
   Each exchange maintains unique opening hours, early closes and holiday schedules. Do not invent artificial K‑lines for times where no market activity ever occurred.

Closing thoughts

The reliability of your backtest output lives or dies by your upstream pre‑processing workflow. Even when you leverage solid market‑data sources like AllTick API, it remains only a gateway to raw market feeds. Whether data is trustworthy enough for strategy testing depends entirely on your own validation, cleansing and tagging logic.

Quant developers often fixate on building clever, sophisticated strategy logic. Practical engineering experience teaches us this lesson: a robust data foundation beats fancy algorithm design every single time. If you resolve market gaps and anomalies early, you eliminate large amounts of hidden backtest bias, and your strategy iteration cycle becomes far less frustrating.

---

💬 Reader mailbag prompt
Have you ever chased false backtest results caused by bad market data? Hit reply and share your story — I may feature anonymised experiences in a future newsletter issue.