How do you fix missing data from Stock APIs in quant backtests?

📝 Tutorial | Engineering

dev.to style: practical, developer‑first, conversational tone, code‑focused. Target audience: backend engineers, quant devs, hobbyists building backtesting tools.

Have you ever run this baffling scenario? Your trading strategy code hasn’t changed one bit, but re‑running your backtest gives completely different results. Signals shift, returns go up or down unexpectedly, drawdowns look nothing like before.

You spend hours auditing indicators, tweaking parameters, hunting for logical bugs in your Python code. Eventually you realize: the problem isn’t your strategy logic. It’s bad historical data coming from stock APIs.

Lots of developers grab K‑line data from an API and throw it straight into their backtesting framework. Fetching data is only the starting point. Timestamps, OHLC prices, and volume need proper validation. This matters even more for minute bars and tick‑level data. One corrupted time slice can cascade and break every downstream indicator calculation.

How missing and malformed data breaks backtests

Quant strategies depend on continuous time‑series market data. Take a simple moving‑average strategy: it calculates values over rolling price windows. When K‑line entries go missing, your calculation window shifts. Buy or sell signals trigger too early or too late, and your whole backtest report loses credibility.

While integrating different stock market APIs, I’ve grouped common data anomalies into four types:

Anomaly Type	Root Cause
Timeline breaks	API response skips records → timestamp jumps
Empty mandatory fields	Null values for price, volume and core trading fields
Duplicate entries	API pushes duplicate market records
Trading‑time misalignment	Time offset caused by different exchange trading rules

If your backtesting code has no safeguards for these issues, simulated performance will diverge from live trading. You may even get inflated, unrealistic profit numbers that will never happen in production.

Pre‑backtest data sanity checks you should implement

When building backtesting pipelines, I treat data validation as a required pre‑processing step before executing any strategy logic.

First, validate timestamps. For minute bars, timestamps must match real exchange trading sessions. For US stocks, timestamps should be continuous during market hours.
When you spot time jumps, distinguish two cases:

1. There were genuinely zero trades in that window
2. The stock API lost some records

These two scenarios require different handling logic.

Next, validate price and volume fields. OHLC and volume feed almost all technical indicators. Any bad value will corrupt your calculations.

You can quickly scan your dataset with Pandas. This snippet is copy‑paste ready for your project:

import pandas as pd

df = pd.read_csv("stock_history.csv")
df["timestamp"] = pd.to_datetime(df["timestamp"])
print(df.isnull().sum())
df = df.sort_values("timestamp")

This script prints null‑value statistics and sorts rows chronologically to catch out‑of‑order timestamps.

Handle missing data differently per data granularity

There is no universal fix for missing market data. Your approach must change based on bar resolution.

For daily bars, small gaps are manageable. Add flag columns to mark gaps and let your strategy skip bad trading days. Avoid overwriting your original raw data.

⚠️ Be extra careful with minute bars and tick streaming data.

High‑frequency backtesting needs strict timeline continuity. If you blindly impute missing prices, you rewrite real‑market behavior and generate misleading, fake backtest outcomes. My team’s approach: keep original records untouched. Add custom status fields to tag rows modified during cleansing for easier debugging and traceability.

For tick‑by‑tick real‑time data, WebSocket connections work better than repeated polling to reduce data loss risk. Below is working example code consuming tick feeds from the AllTick API:

import websocket
import json

def on_message(ws, message):
    data = json.loads(message)
    symbol = data.get("symbol")
    price = data.get("price")
    timestamp = data.get("timestamp")
    print("alltick", symbol, price, timestamp)

ws = websocket.WebSocketApp(
    "wss://api.alltick.co/stock/websocket",
    on_message=on_message
)
ws.run_forever()

💡 Production tip: Receiving raw WebSocket messages is not enough. Add local caching, timestamp checks, and field validation to handle temporary network jitter and prevent data loss.

Common pitfalls developers keep repeating

After iterating on multiple backtesting systems, these three mistakes show up over and over:

1. Don’t fill null values unconditionally
   Strategies have different data‑quality requirements. High‑frequency logic prioritizes raw data integrity. Medium‑long term trend strategies tolerate more noise. Do not reuse one imputation function for every use‑case.

2. Total row count ≠ complete dataset
   Just because you have the expected number of records doesn’t mean your time series is whole. Always cross‑check with exchange calendars and real trading hours.

3. Respect exchange rules when repairing datasets
   Every exchange has unique opening hours, holiday schedules and early closes. Never invent artificial K‑line bars for time periods with zero real‑world market activity.

Wrap‑up

The quality of your backtest output heavily depends on upstream pre‑processing. Even reliable sources such as AllTick API are only a gateway for raw market data. Whether data can be trusted for strategy testing depends entirely on your own validation, cleaning and tagging logic.

Quant developers often spend most time building fancy strategy algorithms. From real‑world experience: a solid data foundation is more valuable than clever algorithm design. Fix market‑data gaps early, and you eliminate lots of hidden backtest bias, making strategy iteration much smoother.