Model A has an average score of 0.02. Model B has 0.01. Is A better?
Maybe. Maybe not. You need two numbers to say.
1. The information coefficient (IC)
For a model that ranks stocks, the IC on a given day is the rank correlation (Spearman) between the model's scores and what each stock did over the following period.
from scipy.stats import spearmanr
ic_by_day = [spearmanr(scores[d], outcome[d]).correlation for d in dates]
mean_ic = np.mean(ic_by_day)
- 0 means no skill: the ranking and the outcome are unrelated.
- Negative means the ranking pointed the wrong way.
- In real markets, an IC around 0.05 held consistently would be strong. Most ideas sit much closer to zero.
2. The t-statistic
The IC bounces around from day to day. The t-statistic asks: is the average big compared with how much it bounces?
Roughly: t = mean_ic / (std_ic / sqrt(n)), but with one catch. Our IC compares rankings with outcomes 20 trading days ahead, so neighbouring days' outcomes overlap and aren't independent. A naive t-statistic overstates confidence. We use a correction for overlapping observations (Newey-West), which makes the t-statistic smaller and more honest.
Rule of thumb: below about 2 means "can't tell".
Our numbers
| Score | Period | IC | t |
|---|---|---|---|
| US v1 | Development | +0.0085 | 0.46 |
| US v1 | Sealed year | −0.0585 | −2.47 |
| India in-v1 | Sealed year | +0.009 | 0.21 |
Most of our models sit in the "can't tell" zone. One result is clearly away from zero, and it's the negative one.
So, is 0.02 better than 0.01?
Only if the gap is large compared with the noise in both. Usually it isn't. Ranking models by their average IC without the t-statistic is ranking them by luck.
From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/method
Educational, not investment advice.
Top comments (0)