How I verify a crypto backtest is real: trade-for-trade parity, bit-for-bit data, and a look-ahead test with teeth

#python #opensource #datascience

Most public trading repos show you a backtest with a nice equity curve. The hard part isn't producing the curve — it's knowing whether the curve is real. So I spent most of my time building the machinery that decides that, and I've open-sourced it: github.com/vinhnguyenthanhdn/ai-crypto

Python 3.10+, MIT. No real orders are ever placed; every execution path is simulated.

Implementation parity, verified trade-for-trade

The single biggest source of false confidence in a backtest is that the thing you tested and the thing you run are not the same program.

So the research reference engines and the production strategy cores are written independently, then reconciled against each other: 963/963, 515/515 and 1,079/1,079 trades matched with zero mismatches, with equity agreeing to 8 decimal places.

That reconciliation runs as a check, not as a one-off claim.

Data verified bit-for-bit

9 years of market history: 942,025 5-minute candles plus 1,508 checksum-verified derivatives archives. Every artifact records a SHA-256 dataset hash, so any result can be tied back to the exact bytes it was computed from.

The price cache was reconciled against the venue API across 3,277 days and matched bit-for-bit.

A look-ahead test that provably has teeth

It's easy to write a look-ahead check that passes because it can't detect anything.

So I deliberately injected look-ahead into a passing contract and measured what happened: training return inflated from +563% to +4,201%. The test can see the bug when the bug is there. That's the part most look-ahead checks never demonstrate.

Costs are verified the same way — measured at −0.29985% on a flat round trip against a stated 0.30%, confirming they're charged exactly once.

Accelerated paper replay through the real lifecycle

3,277 days of history driven through the production SQLite state store, position sizing and accounting. Only the market clock is simulated — everything else is the code that runs live.

Result: 77 ENTRY / 77 EXIT / 77 ledger rows, zero orphaned entries, zero positions left open.

Event-sourced state with full lineage

Append-only event_log and feature_snapshot, plus an idempotent equity_ledger keyed by trade ID.
Position and risk computed from real equity, not notional assumptions.
Every feature snapshot records source exchange, market type, symbol, timeframe, transformation version and the strategy package that consumed it — so any logged decision can be reconstructed from raw candle to logged trade.
Atomic run locking via SQLite BEGIN IMMEDIATE with owner token and heartbeat; a dead process's lease is reclaimed immediately instead of waiting out a stale timeout.
Market-data freshness is enforced: stale collector ticks hard-fail or force a fresh fetch rather than silently reusing an old snapshot.

Costs measured instead of assumed

I sampled 4,695 order book snapshots and pulled base-tier venue fees from official sources. The real half-spread came out at 0.0077 bps, not the 5 bps my own config had assumed — my round-trip cost assumption had been about 3× too high.

Worth knowing if you're calibrating a cost hurdle from a blog post rather than from the book.

Frozen contracts and a documented search

Parameters are selected on the training split only, the contract is frozen, then validation and test are opened once. Every candidate must survive round-trip costs stressed to at least 2× base.

Under that protocol I ran 29,373 parameter configurations across ~40 strategy families, and every one of them is written up — contract, dataset range, cost assumptions, verdict — as ~100 result artifacts. Including the rejections, which is the part that's usually missing and the part that saves you re-walking a dead end.

The repo also audits itself

docs/code-audit.md splits the codebase into a research path that is independently verified and trusted, and a runtime path with confirmed defects — including a BUY threshold that is arithmetically unreachable and a look-ahead bug in the swing detector. Those are catalogued with file and line references rather than quietly patched, so you know which numbers to trust.

The same honesty applies to the headline outcome: none of the 29,373 configurations cleared the promotion gates. Median gross edge across 13,654 rejected configs is −0.021%, so it isn't a cost problem — there's nothing there to rescue. The full evidence is in the repo, and I'd rather publish that than a curve I can't defend.

Try it

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

.venv/bin/python3 scripts/run_backtest.py        # bar-close scan
.venv/bin/python3 scripts/dashboard_server.py    # local dashboard