I run a nightly pipeline that trains scoring models for sixteen stock chart pattern detectors. It has run nearly 800 times since May 2026, and most of its 88,000 lines of Python are not modelling. They are the ruler: the splits, gates and ledgers that decide whether a model is real. The edge it hunts is small (a median out-of-sample rank correlation of 0.031), so telling signal from a lucky draw is the whole job.
The full write-up with the step-by-step chain, the deploy rule and the numbers is on my site. Here are the six ways the pipeline fooled itself, because they generalise beyond finance.
Time leakage. Two detections ten days apart on the same stock share four days of price path. A shuffled k-fold split puts them on opposite sides of a fold and the model learns the future from its neighbours. Fix: contiguous date-block folds with a 16-day purge and a 10-day embargo. A subtler version hid for two months: LightGBM's early-stopping set was the fold it was scored on.
Selection leakage. Fit fifty models and the best looks good whether or not signal exists. Fix: a selection bar that deflates for the number of prior attempts, with the attempt count clustered by prediction similarity so twenty near-identical fits count as fewer than twenty.
Calendar leakage. Features that are nearly constant across stocks on a given day (index correlation, the volatility regime) predicted well in validation because the market rose after days when they were high. Half the validation spread was market timing, and it did not transfer. Fix: a gate on the share of a feature's variance that lies between days, and significance tested over days rather than rows. The honest footnote: live data says this defence is not finished.
Data-draw variance. Two samples of the same detector admitted feature sets that overlapped 52%, and two fitted models shared 29% of their top picks. Variance, not bias, dominates. Fix so far: more rows, and admission by vote across nine seeded fits with shuffled shadow features.
Holdout integrity. The frozen holdout broke five ways before it held, silently each time. The worst: it was keyed by database id, a backtest regenerated the ids, and one detector's freeze matched 0 of 33,013 rows. Fix: natural keys, and a run that refuses to start if fewer than half the keys resolve.
Tooling leaks. A research tool's close_at_rec column was the entry-day close, not the detection-day close. Every feature built on it contained one day of the future. Heuristic: a signal with the same sign on every detector in every period is more likely a leak than a law.
What I would build first if I started again: the self-measurement (planted signals of known size, replays of past runs, audits of projections against realised counts), and the gates second. Full article: https://www.stockdataanalytics.com/blog/how-we-train-stock-pattern-scoring-models/.
Top comments (0)