I build Canli Capital: free, open-source tools that check whether a backtest is real (deflated Sharpe, overfitting and data-snooping tests), plus SEC fundamentals for AI agents over MCP.
Thanks for checking the artifact and for the load-group breakdown. That changes how I'd read the split: 0–22 all sit in one load group, so a bootstrap over scenarios would overstate how much the 23 tell you, and resampling load groups leaves you with one cluster on the unseen side.
If you do another export, there is a contrast that needs no leakage assumption. The 23 (group 0) and the other 70 in the 93 (groups 21, 25, 28) are all unseen. The gap between those two sets is a direct measure of how much RMSE moves with load group alone. If it is about as large as the gap between the 48 seen and the 23 unseen, composition explains the table; if it is much smaller, the seen/unseen difference is left standing. Per-scenario squared-error sums and counts would be enough for both comparisons.
That disjoint 23/70 split is a useful next sensitivity check. I’d report it alongside the 48/23 split, with exact load groups and pooled Pg RMSE computed as sqrt(sum(squared_error) / sum(count)) for each subgroup.
For interpretation, I’d label the 23/70 result “variation among unseen load groups.” A training-exposure estimate would use matched within-load-group samples or a retraining control; the two proposed contrasts involve different load groups.
The current artifacts contain whole-set aggregates, so these subgroup values remain unmeasured. This gives us a concrete next diagnostic if the evaluation is extended. Thanks for following through on the group structure.
I ran the disjoint check with the same pinned checkpoint, saved normalizer, dataset revision and native CPU evaluator—no retraining.
Original membership
Cases / load groups
Pg RMSE (MW)
Seen
48 / 3, 42, 44
8.647
Unseen
23 / 0
18.121
Unseen
70 / 21, 25, 28
15.410
The two unseen sets differ by 2.712 MW (17.6%, relative to unseen-70); their mean objective gaps are 1.639% and 1.145%. That measures variation among these selected unseen load groups. Training history and load-group composition both change in the 48/23 contrast.
There’s also an aggregation wrinkle. The evaluator averages generator squared error inside each batch, then weights batch means by graph count. The fixed raw data has 49–54 active generators per scenario. Recombining squared subgroup RMSE by case count gives 12.527/16.123 MW for 71/93 cases, versus the original native 12.480/16.109 MW. I report each run’s native score; exact global SSE/count pooling needs a per-scenario export.
Thanks for checking the artifact and for the load-group breakdown. That changes how I'd read the split: 0–22 all sit in one load group, so a bootstrap over scenarios would overstate how much the 23 tell you, and resampling load groups leaves you with one cluster on the unseen side.
If you do another export, there is a contrast that needs no leakage assumption. The 23 (group 0) and the other 70 in the 93 (groups 21, 25, 28) are all unseen. The gap between those two sets is a direct measure of how much RMSE moves with load group alone. If it is about as large as the gap between the 48 seen and the 23 unseen, composition explains the table; if it is much smaller, the seen/unseen difference is left standing. Per-scenario squared-error sums and counts would be enough for both comparisons.
That disjoint 23/70 split is a useful next sensitivity check. I’d report it alongside the 48/23 split, with exact load groups and pooled Pg RMSE computed as
sqrt(sum(squared_error) / sum(count))for each subgroup.For interpretation, I’d label the 23/70 result “variation among unseen load groups.” A training-exposure estimate would use matched within-load-group samples or a retraining control; the two proposed contrasts involve different load groups.
The current artifacts contain whole-set aggregates, so these subgroup values remain unmeasured. This gives us a concrete next diagnostic if the evaluation is extended. Thanks for following through on the group structure.
I ran the disjoint check with the same pinned checkpoint, saved normalizer, dataset revision and native CPU evaluator—no retraining.
The two unseen sets differ by 2.712 MW (17.6%, relative to unseen-70); their mean objective gaps are 1.639% and 1.145%. That measures variation among these selected unseen load groups. Training history and load-group composition both change in the 48/23 contrast.
There’s also an aggregation wrinkle. The evaluator averages generator squared error inside each batch, then weights batch means by graph count. The fixed raw data has 49–54 active generators per scenario. Recombining squared subgroup RMSE by case count gives 12.527/16.123 MW for 71/93 cases, versus the original native 12.480/16.109 MW. I report each run’s native score; exact global SSE/count pooling needs a per-scenario export.
Report, exact IDs and CSVs · Completed run
Thanks for suggesting the disjoint comparison. The report credits your diagnostic and my AI engineering contribution.