Before any backtest there is the data problem, and it's where most retail strategies die quietly: the free feed ships a silent error, the backtest eats it, and the equity curve lies. Here's how to pull usable OHLCV data for stocks and crypto in Python with $0 in data fees — and how to catch the errors free sources actually ship with.
Part 2 of this series: How to Clean Messy OHLCV Data in Python Without Writing a Custom Validator — the cleaning pass that makes this data honest before you backtest on it.
1. Stocks and ETFs: yfinance is still the fastest start
import yfinance as yf
df = yf.download("AAPL", start="2020-01-01", auto_adjust=True)
Two settings that matter: auto_adjust=True gives you split/dividend-adjusted prices (backtest on adjusted, always — unadjusted close will fake a -50% drawdown on every split), and Yahoo rate-limits aggressively, so batch tickers in one call rather than looping with sleeps.
2. Crypto: ccxt against any major exchange, no API key
import ccxt
ex = ccxt.binance()
ohlcv = ex.fetch_ohlcv("BTC/USDT", "1d", since=ex.parse8601("2020-01-01T00:00:00Z"), limit=1000)
Public market data needs no key on most exchanges. The one trap: fetch_ohlcv returns at most limit candles, so you must paginate with since = last timestamp + 1 until you reach today. Many "my backtest starts mid-2021" bugs are just missing pagination.
3. Trust nothing: run the four sanity checks
Free data ships with errors that look like signals:
-
Duplicates — same timestamp twice (re-runs, API hiccups).
df = df[~df.index.duplicated()]. - Gaps — a missing day is usually a market holiday, not missing data; forward-filling it invents fake prices. Crypto trades 24/7, so any gap there is a real data hole.
- OHLC violations — high must be ≥ open/close/low, low must be ≤ open/close. Any violation means corrupted rows; drop them.
- Zero-volume ghosts — long runs of volume=0 with moving prices are stale candles from the exchange, not a quiet market.
4. One timezone, one calendar
Mixing UTC crypto candles with exchange-local stock bars shifts every daily signal by hours. Convert everything to UTC (or exchange-native) once at ingestion, and never compare a daily bar computed from different session boundaries across assets.
5. Survivorship bias: the error you can't fix for free
Free sources list what trades today. Backtest a "buy the dip" strategy on today's S&P 500 constituents and you've silently excluded every company that died. For daily-frequency retail strategies the bias is tolerable if you know it's there — fatal if you don't. Free fix: note the caveat in your results; real fix: point-in-time constituent files, which are usually paid.
6. Store as Parquet, not CSV
CSV round-trips turn your datetimes into strings and your NaNs into empty cells. One line: df.to_parquet("btc_1d.parquet"). Types survive, files shrink ~5x, loads get faster — and your backtest stops re-parsing dates every run.
7. Re-pull, don't accumulate
Free endpoints get revised (Yahoo adjusts history after splits). A local cache that grows append-only will silently diverge from what the exchange says happened. Re-download the full range weekly; keep the cache for intraday iteration only.
Steps 3, 4 and 6 are exactly what the data cleaner in my Python Backtest Toolkit automates: point it at a messy CSV/Excel export and it runs the duplicate/gap/OHLC/volume checks, fixes timezones, and writes clean Parquet ready for a backtest. The toolkit also ships a one-command backtrader template (Sharpe, max drawdown, win rate), 3 sample strategies and a 500-day sample OHLCV set. $1 launch price while collecting the first honest reviews — if the cleaner misses an error type you've hit, tell me and I'll add it: https://datafixtools.sell.app/product/python-backtest-toolkit
Not investment advice. Educational tooling only.
Top comments (0)