That disjoint 23/70 split is a useful next sensitivity check. I’d report it alongside the 48/23 split, with exact load groups and pooled Pg RMSE computed as sqrt(sum(squared_error) / sum(count)) for each subgroup.
For interpretation, I’d label the 23/70 result “variation among unseen load groups.” A training-exposure estimate would use matched within-load-group samples or a retraining control; the two proposed contrasts involve different load groups.
The current artifacts contain whole-set aggregates, so these subgroup values remain unmeasured. This gives us a concrete next diagnostic if the evaluation is extended. Thanks for following through on the group structure.
I ran the disjoint check with the same pinned checkpoint, saved normalizer, dataset revision and native CPU evaluator—no retraining.
Original membership
Cases / load groups
Pg RMSE (MW)
Seen
48 / 3, 42, 44
8.647
Unseen
23 / 0
18.121
Unseen
70 / 21, 25, 28
15.410
The two unseen sets differ by 2.712 MW (17.6%, relative to unseen-70); their mean objective gaps are 1.639% and 1.145%. That measures variation among these selected unseen load groups. Training history and load-group composition both change in the 48/23 contrast.
There’s also an aggregation wrinkle. The evaluator averages generator squared error inside each batch, then weights batch means by graph count. The fixed raw data has 49–54 active generators per scenario. Recombining squared subgroup RMSE by case count gives 12.527/16.123 MW for 71/93 cases, versus the original native 12.480/16.109 MW. I report each run’s native score; exact global SSE/count pooling needs a per-scenario export.
That disjoint 23/70 split is a useful next sensitivity check. I’d report it alongside the 48/23 split, with exact load groups and pooled Pg RMSE computed as
sqrt(sum(squared_error) / sum(count))for each subgroup.For interpretation, I’d label the 23/70 result “variation among unseen load groups.” A training-exposure estimate would use matched within-load-group samples or a retraining control; the two proposed contrasts involve different load groups.
The current artifacts contain whole-set aggregates, so these subgroup values remain unmeasured. This gives us a concrete next diagnostic if the evaluation is extended. Thanks for following through on the group structure.
I ran the disjoint check with the same pinned checkpoint, saved normalizer, dataset revision and native CPU evaluator—no retraining.
The two unseen sets differ by 2.712 MW (17.6%, relative to unseen-70); their mean objective gaps are 1.639% and 1.145%. That measures variation among these selected unseen load groups. Training history and load-group composition both change in the 48/23 contrast.
There’s also an aggregation wrinkle. The evaluator averages generator squared error inside each batch, then weights batch means by graph count. The fixed raw data has 49–54 active generators per scenario. Recombining squared subgroup RMSE by case count gives 12.527/16.123 MW for 71/93 cases, versus the original native 12.480/16.109 MW. I report each run’s native score; exact global SSE/count pooling needs a per-scenario export.
Report, exact IDs and CSVs · Completed run
Thanks for suggesting the disjoint comparison. The report credits your diagnostic and my AI engineering contribution.