A controlled study of 51 post-training algorithms across 240 runs finds algorithm performance rankings completely invert between 1.5B and 7B parameter models. The choice of loss function provides less than 1 percentage point of leverage compared to model scale.

OXRL Study: Post-Training Algorithm Rankings Invert with Model Scale, Loss Modifications Offer Negligible Gains

A comprehensive, controlled study from researchers at arXiv has delivered a sobering reality check for the post-training alignment community. The paper, "Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversions," presents the results of the OXRL framework—a unified system that implemented 51 different post-training algorithms with identical infrastructure to enable the first true apples-to-apples comparison.

The study, which required approximately 240 training runs on H100 GPUs, systematically evaluated 8 core algorithms across 4 model scales (0.5B to 7B parameters), 3 evaluation domains, and a taxonomy of 20 DPO variants. The findings challenge several assumptions about post-training algorithm selection and reveal surprising scale-dependent behaviors that could reshape how practitioners approach model alignment.

What the Researchers Built: The OXRL Framework

The core contribution of this work is OXRL (Open eXperimental Reinforcement Learning), a unified framework designed specifically for controlled comparisons. Previous evaluations of post-training algorithms suffered from inconsistent implementations, varying hyperparameters, and different evaluation protocols, making it difficult to determine whether observed performance differences stemmed from algorithmic advantages or implementation details.

OXRL solves this by providing identical infrastructure for all 51 algorithms it implements, including:

• Consistent data loading and preprocessing pipelines
• Identical training schedules and optimization settings
• Standardized evaluation protocols across all benchmarks
• Reproducible seeding and statistical analysis methods

This infrastructure enabled the researchers to run 100 separate DPO variant experiments at the 1.5B parameter scale alone, with 5 random seeds each, ensuring statistical significance in their comparisons.

Key Results: Scale-Dependent Ranking Inversions

The most striking finding from the study is that algorithm performance rankings are not stable across different model scales. What works best for smaller models may perform worst for larger ones, and vice versa.

At 1.5B parameters on GSM8K:

• Online RL (SGRPO) achieved the best performance: 58.0% ± 0.57
• SimPO performed worst among the methods tested

At 7B parameters on GSM8K:

• SimPO became the best performer: 85.8%
• The previously best-performing SGRPO was no longer optimal

This represents a complete ranking inversion driven solely by model scale. The researchers confirmed this effect was due to scale itself rather than other factors like LoRA regularization through a 2×2 factorial design that isolated the variables.

Performance Comparison Table (GSM8K Benchmark)

Model Scale	Best Algorithm	Performance	Worst Algorithm	Performance	Performance Spread
1.5B	SGRPO (Online RL)	58.0%	SimPO	~38.7%	19.3 pp
7B	SimPO	85.8%	(Various)	~77%	~8.8 pp

How It Works: Methodology and Experimental Design

The study employed a rigorous experimental design to isolate the effects of different variables:

Model Scales: Four parameter counts were tested: 0.5B, 1.5B, 3B, and 7B, allowing the researchers to trace performance trends across the scaling curve.

Algorithm Categories: The study evaluated eight core algorithm families:

1. Direct Preference Optimization (DPO) and its 20 variants
2. SimPO (Simple Preference Optimization)
3. KTO (Kahneman-Tversky Optimization)
4. GRPO (Group Relative Policy Optimization)
5. SGRPO (Stochastic GRPO)
6. Online RL methods
7. Offline RL methods
8. Hybrid approaches

Evaluation Domains: Three distinct evaluation areas were used:

1. GSM8K: Grade school math problems (in-distribution for training)
2. MATH: More challenging mathematical reasoning (out-of-distribution)
3. General-domain benchmarks: Standard LLM evaluation suites

Statistical Rigor: All comparisons used Bonferroni correction for multiple hypothesis testing, with p-values reported for significant findings. The 5-seed repetition for each configuration provided robust error estimates.

Why It Matters: A Hierarchy of Leverage for Practitioners

The study's most practical contribution is a clear hierarchy of what actually matters when aligning language models:

1. Model Scale (~50 percentage points): The single most important factor. Moving from 0.5B to 7B parameters provides approximately 50 percentage points of improvement on GSM8K, dwarfing all algorithmic differences.

2. Training Paradigm (~10 pp): The choice between different algorithm families (DPO vs. RL vs. hybrid) provides meaningful but smaller gains.

3. Online vs. Offline (~9 pp): Whether the algorithm uses online data collection or offline preference data creates noticeable differences.

4. Loss Function Modifications (~1 pp): Surprisingly, none of the 20 DPO variants significantly outperformed vanilla DPO after statistical correction. The modifications that researchers have proposed—changing margins, adding regularizers, or adjusting temperature—provide negligible practical benefit.

The sole exception was SimPO, which performed significantly worse than vanilla DPO at smaller scales (-11.5 percentage points, p < 10⁻⁴) but became optimal at larger scales.

Task-Specific Algorithm Leverage

Another critical finding is that algorithm choice matters primarily within the training distribution. The 19.3 percentage point spread between best and worst algorithms on GSM8K (which matches the training distribution) collapses to:

• 0.54 percentage points on MATH (a 36× reduction)
• 0.47 percentage points on general-domain benchmarks (a 41× reduction)

This suggests that while algorithm selection can significantly impact performance on target tasks, it provides minimal benefit for general capabilities or out-of-distribution generalization.

gentic.news Analysis

This study represents a watershed moment for the post-training alignment field. For years, researchers have proposed countless algorithmic variants—each claiming incremental improvements over DPO—without rigorous, controlled comparisons. The OXRL framework finally provides the methodological rigor needed to separate signal from noise.

The scale-dependent ranking inversions are particularly significant. They suggest that the community has been optimizing for the wrong thing: much of the published research focuses on smaller models due to computational constraints, but findings from 1-3B parameter models may not translate to the 7-70B scale where most production systems operate. This could explain why some algorithms that show promise in research papers fail to deliver in practical applications.

The negligible impact of loss function modifications is equally important. The DPO literature has exploded with variants claiming theoretical advantages, but this study suggests most provide no practical benefit. This should prompt a reevaluation of research priorities: perhaps the field should focus less on minor loss function tweaks and more on fundamental issues like data quality, training stability, and compute-efficient methods.

For practitioners, the clear hierarchy of leverage provides actionable guidance: invest in larger models first, choose a reasonable algorithm family second, and don't waste time optimizing loss function details. The released OXRL framework and all associated code, configs, and evaluation data will serve as a valuable community benchmark, potentially becoming the standard for future algorithm comparisons.

Frequently Asked Questions

What is the most important factor in post-training alignment according to this study?

Model scale is overwhelmingly the most important factor, providing approximately 50 percentage points of improvement on the GSM8K benchmark when moving from 0.5B to 7B parameters. This dwarfs the impact of algorithm choice, which provides at most 10-20 percentage points of difference. The study's hierarchy of leverage clearly shows: scale > training paradigm > online/offline > loss function modifications.

Do DPO variants actually improve performance over vanilla DPO?

No, not in any statistically significant way. After applying Bonferroni correction for multiple hypothesis testing, none of the 20 tested DPO variants significantly outperformed vanilla DPO. The sole exception was SimPO, which actually performed significantly worse at smaller scales (-11.5 percentage points) before becoming optimal at larger scales. This suggests that most proposed modifications to the DPO loss function provide negligible practical benefit.

Why does algorithm performance ranking invert with model scale?

The study found that algorithms optimal for smaller models (like SGRPO at 1.5B) become suboptimal at larger scales (7B), while algorithms that perform poorly at small scales (like SimPO) become optimal at larger scales. The researchers confirmed this is driven by model scale itself through controlled factorial experiments. This scale-dependence suggests that findings from research on smaller models may not generalize to the larger scales used in production systems.

How much does algorithm choice matter for out-of-distribution performance?

Very little. While algorithm selection created a 19.3 percentage point spread on GSM8K (which matches the training distribution), this spread collapsed to just 0.54 percentage points on the more challenging MATH benchmark and 0.47 percentage points on general-domain benchmarks. This means algorithm choice primarily affects performance on the specific tasks you're training for, with minimal impact on general capabilities or out-of-distribution generalization.

---

Originally published on gentic.news