Here's a loop most of us have run:
- Build a model.
- Check it on the test set.
- Not great. Tweak a feature.
- Check again.
After twenty rounds the test score looks good. But you've been fitting the test set, one decision at a time. The number no longer estimates how the model does on new data.
Sealing it
For our stock score we did something stricter:
- Development data (2020–2025): build, explore, change things freely.
- Final year: sealed. Nobody looks at it.
-
Freeze: every rule written down and versioned (
v1) before the seal is broken: features, weights, the universe, the measure, the significance test. - Open once. Report whatever comes out.
Ours came out negative (IC −0.0585, t −2.47). We published it, and we did not try a "v1.1" on the same year. A new version has to be announced first and tested on data that arrives after the announcement.
What counts as a peek
More than you'd think:
- Plotting the holdout to "sanity-check" it.
- Choosing a hyperparameter grid after hearing how a colleague's model did on it.
- Re-running with a different random seed because the first one looked unlucky.
- Dropping a stock from the holdout because "its data looked odd".
Each one leaks a little information from the test set into your choices.
A cheap version for your own projects
You don't need special tooling:
- Put the holdout in a separate file, and don't load it in your notebooks.
- Write the frozen config to a file and commit it (a hash in git is a timestamp).
- Write the evaluation script before you run it, and run it once.
- Record the result even if you hate it.
It's uncomfortable. It's also the only way your test number means what you think it means.
From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/method
Educational, not investment advice.
Top comments (0)