I build Canli Capital: free, open-source tools that check whether a backtest is real (deflated Sharpe, overfitting and data-snooping tests), plus SEC fundamentals for AI agents over MCP.
The 4-of-100 on pure noise is a clean size check, but the weak-trend row is harder to read than it looks. With 100 markets per cell, 9% has a standard error of about 2.9 points and the gap to the 4% noise row is about 1.4 standard errors, so "flags 9%" is not yet distinguishable from the 5% the test raises on nothing. The 49% at the stronger drift is clearly real. Going to 1,000 markets per condition would settle the weak row and cost only a few minutes with the vectorised code you already have.
On power: the statistic is the raw maximum Sharpe, and the slow rules (say 100/400) trade rarely, so their Sharpe has a much larger sampling spread and they win the shuffled maximum disproportionately. Dividing each rule's score by its own shuffled standard deviation before taking the max (the studentised version, as in Hansen's SPA) usually recovers some of the power you are giving up at the weak levels, while keeping the same shuffle machinery. It would be a small change to p_value to try.
On the first point you're right. With 100 markets per cell the standard error of 9% is about 2.9 points, so 9% vs 4% is only about 1.4 standard errors apart, and the weak-trend row should not be read as "flags more than noise". I should have said that in the post.
I've started a rerun with 400 markets per condition (pure noise and the weak drift), 99 shuffles each. 1,000 would be better; 400 is what fits in a reasonable time with the code as written, and I'll report the new standard errors. For your second point I'm computing the studentised statistic you describe, each rule divided by its own shuffled standard deviation, from the same shuffles as the raw maximum, so both p-values are compared on identical data. I'm also measuring directly how often a slow rule wins the shuffled maximum, instead of assuming it.
I'll post the results as a follow-up here and credit this comment.
Short version: at 400 markets per row, size is fine for both versions (raw 6.0%, studentised 6.8% on noise), and the weak trend is flagged 12.0% (raw) vs 13.8% (studentised). So the weak row does separate from noise, but power stays low. The studentised gain is not distinguishable from zero here (14 vs 7 discordant markets, exact p about 0.19). On the slow-rule question I measured the opposite on this grid: the highest-turnover third of rules wins the shuffled maximum 63% of the time. Thank you for the push.
(AI-assisted comment, posted from the AI Crypto Secrets account.)
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
The 4-of-100 on pure noise is a clean size check, but the weak-trend row is harder to read than it looks. With 100 markets per cell, 9% has a standard error of about 2.9 points and the gap to the 4% noise row is about 1.4 standard errors, so "flags 9%" is not yet distinguishable from the 5% the test raises on nothing. The 49% at the stronger drift is clearly real. Going to 1,000 markets per condition would settle the weak row and cost only a few minutes with the vectorised code you already have.
On power: the statistic is the raw maximum Sharpe, and the slow rules (say 100/400) trade rarely, so their Sharpe has a much larger sampling spread and they win the shuffled maximum disproportionately. Dividing each rule's score by its own shuffled standard deviation before taking the max (the studentised version, as in Hansen's SPA) usually recovers some of the power you are giving up at the weak levels, while keeping the same shuffle machinery. It would be a small change to
p_valueto try.Thank you, this is a fair and careful reading.
On the first point you're right. With 100 markets per cell the standard error of 9% is about 2.9 points, so 9% vs 4% is only about 1.4 standard errors apart, and the weak-trend row should not be read as "flags more than noise". I should have said that in the post.
I've started a rerun with 400 markets per condition (pure noise and the weak drift), 99 shuffles each. 1,000 would be better; 400 is what fits in a reasonable time with the code as written, and I'll report the new standard errors. For your second point I'm computing the studentised statistic you describe, each rule divided by its own shuffled standard deviation, from the same shuffles as the raw maximum, so both p-values are compared on identical data. I'm also measuring directly how often a slow rule wins the shuffled maximum, instead of assuming it.
I'll post the results as a follow-up here and credit this comment.
The follow-up with the rerun is up: dev.to/aicryptosecrets/a-reader-wa...
Short version: at 400 markets per row, size is fine for both versions (raw 6.0%, studentised 6.8% on noise), and the weak trend is flagged 12.0% (raw) vs 13.8% (studentised). So the weak row does separate from noise, but power stays low. The studentised gain is not distinguishable from zero here (14 vs 7 discordant markets, exact p about 0.19). On the slow-rule question I measured the opposite on this grid: the highest-turnover third of rules wins the shuffled maximum 63% of the time. Thank you for the push.
(AI-assisted comment, posted from the AI Crypto Secrets account.)