DEV Community

findatacity
findatacity

Posted on

What 18M rows of A-share daily bars taught me about "clean data"

What 18M rows of A-share daily bars taught me about "clean data"

Three backtests died before I admitted the obvious: my data was dirty, and no amount of feature engineering was going to save it.

This is a write-up of the three failure modes I hit, the checks that now catch them, and the schema I ended up with. Every number below is reproducible from the CSVs I ship — there's a RECOMPUTE.py in the dataset that regenerates all of it.

Failure 1: the adjustment seam

Unadjusted prices jump at every split and every dividend. If your return series crosses one of those jumps, you get a fake −50% day and your drawdown, volatility and Sharpe are all garbage.

The naive fix — always use adjusted prices — has its own trap: adjusted series get rewritten when a new corporate action happens, so a backtest written last year silently produces different results today.

What I do instead: keep raw, unadjusted OHLCV exactly as the exchange publishes it, and keep the adjustment factors in a separate file with the event dates. The backtest decides which view it wants, and the choice is explicit and auditable.

Check that catches it:

# flag any single-day move that looks like a corporate action, not a market move
rets = [c[i]/c[i-1]-1 for i in range(1, len(c))]
suspect = [i for i, r in enumerate(rets) if abs(r) > 0.25]
Enter fullscreen mode Exit fullscreen mode

If a "−50% day" shows up in a large-cap index, it's almost never the market. It's a seam.

Failure 2: phantom bars on halted days

Some sources fill suspended days by forward-filling the last close, or worse, by writing a row with zero volume and the previous close. A backtest then happily trades a stock that couldn't be traded.

My rule: a halt is an absent row, not a synthetic one. If the exchange didn't trade, there is no bar. That makes joins sparser but keeps the backtest honest.

The corollary matters for indices: an index can print a value on a day when half its constituents are halted. That's fine — the index is the index. But the constituent table must not invent fills to make the row count look tidy.

Failure 3: the ticker that changed its name

Chinese A-shares rename constantly — restructuring prefixes (G, S, ST, *ST) get added and stripped, and companies change names outright. If you key your universe on the ticker string, a rename silently splits one company into two, and a delisting silently removes a loser from your sample.

That last one is survivorship bias, and it flatters every strategy. The fix is boring: keep a former-name chain and an explicit delisted flag, and never drop a row because the company stopped existing.

Concretely, the A-share sample I publish includes 5,923 tickers and 17,964,412 daily rows, delisted names included. It also carries listing dates and industry classification so you don't have to reconstruct them from tickers.

The schema I ended up with

Seven columns, one CSV per series, UTF-8:

column meaning
date YYYY-MM-DD, exchange local
open raw unadjusted
close raw unadjusted
high raw unadjusted
low raw unadjusted
volume shares / contracts
amount turnover

Two things people trip over:

  1. close comes before high/low. That's how the source API returns it. I kept the order and documented it instead of silently reordering — a loader that assumes OHLC will swap two columns and produce nonsense. FIELD.md in the dataset says this in the first line.
  2. amount = 0.0 means "the source doesn't publish turnover", not "zero turnover". Several index futures series have this. Encoding "unknown" as 0 is a bug waiting to happen, so it's documented rather than guessed.

Numbers, so you can check me

From the 8 A-share broad indices (44,577 daily bars total, 1990 → 2026):

  • SSE Composite: 8,758 bars, max drawdown −78.27% (1993–94), CAGR 10.91% from the first published close
  • CSI 300: 5,304 bars, max drawdown −72.30%
  • STAR 50: 1,659 bars starting 2019-12-31 — and no fabricated history before its launch date

Run RECOMPUTE.py on the shipped files and you get exactly these numbers. That's the point: if a data seller won't ship the recompute script, you can't tell whether the numbers are real.

Where to get the samples

16 free series (8 A-share indices, 3 US indices, 5 Chinese futures main-continuous products), CC BY 4.0, no signup:

The paid packs (5,923 A-share tickers, 13,400 US tickers, 81 futures products with contract-level detail) exist, but the free samples are enough to check whether the schema fits your pipeline.

One question for you

How do you handle suspended days — absent rows, forward-fill, or last-close-with-zero-volume? I've seen all three in production pipelines and I'm genuinely curious which one survives contact with your backtests.


I build cleaned market datasets at findatacity.com. Numeric claims in this post are reproducible from the shipped CSVs; the recompute script is in the dataset.

Top comments (0)