What 18M rows of A-share daily bars taught me about "clean data"
Three backtests died before I admitted the obvious: my data was dirty, and no amount of feature engineering was going to save it.
This is a write-up of the three failure modes I hit, the checks that now catch them, and the schema I ended up with. Every number below is reproducible from the CSVs I ship — there's a RECOMPUTE.py in the dataset that regenerates all of it.
Failure 1: the adjustment seam
Unadjusted prices jump at every split and every dividend. If your return series crosses one of those jumps, you get a fake −50% day and your drawdown, volatility and Sharpe are all garbage.
The naive fix — always use adjusted prices — has its own trap: adjusted series get rewritten when a new corporate action happens, so a backtest written last year silently produces different results today.
What I do instead: keep raw, unadjusted OHLCV exactly as the exchange publishes it, and keep the adjustment factors in a separate file with the event dates. The backtest decides which view it wants, and the choice is explicit and auditable.
Check that catches it:
# flag any single-day move that looks like a corporate action, not a market move
rets = [c[i]/c[i-1]-1 for i in range(1, len(c))]
suspect = [i for i, r in enumerate(rets) if abs(r) > 0.25]
If a "−50% day" shows up in a large-cap index, it's almost never the market. It's a seam.
Failure 2: phantom bars on halted days
Some sources fill suspended days by forward-filling the last close, or worse, by writing a row with zero volume and the previous close. A backtest then happily trades a stock that couldn't be traded.
My rule: a halt is an absent row, not a synthetic one. If the exchange didn't trade, there is no bar. That makes joins sparser but keeps the backtest honest.
The corollary matters for indices: an index can print a value on a day when half its constituents are halted. That's fine — the index is the index. But the constituent table must not invent fills to make the row count look tidy.
Failure 3: the ticker that changed its name
Chinese A-shares rename constantly — restructuring prefixes (G, S, ST, *ST) get added and stripped, and companies change names outright. If you key your universe on the ticker string, a rename silently splits one company into two, and a delisting silently removes a loser from your sample.
That last one is survivorship bias, and it flatters every strategy. The fix is boring: keep a former-name chain and an explicit delisted flag, and never drop a row because the company stopped existing.
Concretely, the A-share sample I publish includes 5,923 tickers and 17,964,412 daily rows, delisted names included. It also carries listing dates and industry classification so you don't have to reconstruct them from tickers.
The schema I ended up with
Seven columns, one CSV per series, UTF-8:
| column | meaning |
|---|---|
| date | YYYY-MM-DD, exchange local |
| open | raw unadjusted |
| close | raw unadjusted |
| high | raw unadjusted |
| low | raw unadjusted |
| volume | shares / contracts |
| amount | turnover |
Two things people trip over:
-
closecomes beforehigh/low. That's how the source API returns it. I kept the order and documented it instead of silently reordering — a loader that assumes OHLC will swap two columns and produce nonsense.FIELD.mdin the dataset says this in the first line. -
amount = 0.0means "the source doesn't publish turnover", not "zero turnover". Several index futures series have this. Encoding "unknown" as 0 is a bug waiting to happen, so it's documented rather than guessed.
Numbers, so you can check me
From the 8 A-share broad indices (44,577 daily bars total, 1990 → 2026):
- SSE Composite: 8,758 bars, max drawdown −78.27% (1993–94), CAGR 10.91% from the first published close
- CSI 300: 5,304 bars, max drawdown −72.30%
- STAR 50: 1,659 bars starting 2019-12-31 — and no fabricated history before its launch date
Run RECOMPUTE.py on the shipped files and you get exactly these numbers. That's the point: if a data seller won't ship the recompute script, you can't tell whether the numbers are real.
Where to get the samples
16 free series (8 A-share indices, 3 US indices, 5 Chinese futures main-continuous products), CC BY 4.0, no signup:
- Direct CSV: https://findatacity.com/data/
- Kaggle: https://www.kaggle.com/datasets/zhaoyaosakura/ashare-index-daily-bars
The paid packs (5,923 A-share tickers, 13,400 US tickers, 81 futures products with contract-level detail) exist, but the free samples are enough to check whether the schema fits your pipeline.
One question for you
How do you handle suspended days — absent rows, forward-fill, or last-close-with-zero-volume? I've seen all three in production pipelines and I'm genuinely curious which one survives contact with your backtests.
I build cleaned market datasets at findatacity.com. Numeric claims in this post are reproducible from the shipped CSVs; the recompute script is in the dataset.
Top comments (0)