A model that ranks stocks with an information coefficient (IC) of 0.02 sounds like something. Is it? The quickest way to find out is to compare it with things that can't have skill.
We run two kinds of random referee next to every result.
1. Shuffled rankings (is the result better than chance?)
Keep everything about the strategy the same (number of stocks, rebalancing dates, costs), but pick names at random. Do it many times. You get a distribution of results that pure luck produces with your exact setup.
Two flavours are useful:
- Cross-sectional shuffle: each day, shuffle scores across stocks. Tests whether the ranking itself carries information.
- Time-block shuffle: shuffle whole blocks of dates. Keeps the time structure, so it catches results that come from one lucky period.
If your model sits inside the cloud of random results, it hasn't shown skill, whatever the headline number.
2. Shuffled labels (does the pipeline leak?)
Train the full model, with the same features, the same walk-forward schedule, the same everything, on labels that have been randomly shuffled. There is nothing real to learn. The IC should come out around zero.
If it doesn't, something in your pipeline is leaking the answer: a feature computed with future data, an overlapping window, a join on the wrong date.
For our LightGBM model, the shuffled-label version scored about zero, which is what you want to see.
What we found
Our score and the twelve model families we tried didn't clearly beat the random referees. Seeing that side by side is more useful than any single number: it shows how big "nothing" looks in this data.
Try it
Before you trust any ranking model, ask two questions: how does it compare with random picks, and what does it score on shuffled labels? If nobody ran those, the result isn't finished.
From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/research
Educational, not investment advice.
Top comments (0)