I ran the disjoint check with the same pinned checkpoint, saved normalizer, dataset revision and native CPU evaluator—no retraining.
Original membership
Cases / load groups
Pg RMSE (MW)
Seen
48 / 3, 42, 44
8.647
Unseen
23 / 0
18.121
Unseen
70 / 21, 25, 28
15.410
The two unseen sets differ by 2.712 MW (17.6%, relative to unseen-70); their mean objective gaps are 1.639% and 1.145%. That measures variation among these selected unseen load groups. Training history and load-group composition both change in the 48/23 contrast.
There’s also an aggregation wrinkle. The evaluator averages generator squared error inside each batch, then weights batch means by graph count. The fixed raw data has 49–54 active generators per scenario. Recombining squared subgroup RMSE by case count gives 12.527/16.123 MW for 71/93 cases, versus the original native 12.480/16.109 MW. I report each run’s native score; exact global SSE/count pooling needs a per-scenario export.
I ran the disjoint check with the same pinned checkpoint, saved normalizer, dataset revision and native CPU evaluator—no retraining.
The two unseen sets differ by 2.712 MW (17.6%, relative to unseen-70); their mean objective gaps are 1.639% and 1.145%. That measures variation among these selected unseen load groups. Training history and load-group composition both change in the 48/23 contrast.
There’s also an aggregation wrinkle. The evaluator averages generator squared error inside each batch, then weights batch means by graph count. The fixed raw data has 49–54 active generators per scenario. Recombining squared subgroup RMSE by case count gives 12.527/16.123 MW for 71/93 cases, versus the original native 12.480/16.109 MW. I report each run’s native score; exact global SSE/count pooling needs a per-scenario export.
Report, exact IDs and CSVs · Completed run
Thanks for suggesting the disjoint comparison. The report credits your diagnostic and my AI engineering contribution.