Are We Optimizing the Training Process—or the Model’s Ability to Reason?

In large language model reinforcement learning, better training metrics do not always translate into better reasoning performance.

A lower training loss, higher reward, or continuously updated policy may look like progress. But when the model is evaluated in a real inference environment, the improvement can be much smaller—or may not appear at all.

Why does this happen?

One possible explanation is the mismatch between training and inference. Differences in sampling strategies, numerical precision, inference engines, and system configurations can cause the model to behave differently during training and deployment.

This raises a fundamental question:

Should we optimize the training policy itself, or should we directly optimize how the model performs during inference?

MemoraX AI, together with Tianjin University and Alibaba, explores this question in a research project for NeurIPS 2026:

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

Paper:

https://arxiv.org/abs/2606.29526

Project page:

https://anitaleungxx.github.io/MIPU

The problem: training improvements do not always transfer to inference

Most reinforcement learning pipelines focus on improving the training process through reward design, policy updates, sampling strategies, and optimization objectives.

However, the training environment and the inference environment are not always identical.

Training may involve systems such as vLLM, SGLang, Megatron, or FSDP, while inference may use a different runtime or sampling configuration. Even small differences in implementation can affect the model’s output distribution.

As a result, a policy that looks better during training may not necessarily produce better answers during inference.

This training–inference mismatch can lead to several problems:

• Training rewards may not fully reflect real-world reasoning quality;
• Some policy updates may improve local metrics without improving final answers;
• Model performance may fluctuate during training;
• Improvements may not transfer consistently across inference systems.

The key question is therefore not simply whether the training policy has improved.

The more important question is:

Did the model actually become better at solving the task?

MIPI: making inference quality the real objective

MIPI starts from a simple idea:

Training updates should be evaluated by how they affect the model’s actual inference behavior.

Let μₖ denote the inference policy after the k-th update, πₖ denote the training policy at step k, and J denote the model’s performance on the target inference task.

MIPI aims to enforce:

J(μₖ₊₁) − J(μₖ) ≥ 0

In other words, each effective update should preserve or improve the model’s inference performance.

The change in inference performance can be decomposed as:

J(μₖ₊₁) − J(μₖ)

= [J(μₖ₊₁) − J(πₖ₊₁)]

+ [J(πₖ₊₁) − J(πₖ)]

+ [J(πₖ) − J(μₖ)]

This decomposition highlights three different factors:

1. The gap between the updated inference policy and the updated training policy;
2. The improvement introduced by the training update itself;
3. The gap between the previous training policy and the previous inference policy.

This perspective suggests that improving the training policy alone is not enough. We also need to understand how training updates affect the model’s actual inference behavior.

MIPU: turning the principle into a practical method

To make MIPI actionable, the paper introduces MIPU.

MIPU consists of two main stages:

1. Generate candidate updates;
2. Evaluate candidate inference behavior and decide whether to accept the update.

Step 1: Generate candidate updates

MIPU compares the behavior of the policy before and after an update.

It considers the probability assigned to candidate responses by both the current training policy and the updated policy:

πᵏθ / μₖ = (πₖ / μₖ) × (πθ / πₖ)

This allows MIPU to separate changes caused by the training update from differences between training-time and inference-time behavior.

The first stage produces a set of candidate models or candidate updates for further evaluation.

Step 2: Evaluate candidate responses

MIPU then evaluates the actual responses generated by each candidate.

An update is more valuable when it:

• Increases the probability of high-quality answers;
• Reduces the probability of low-quality answers;
• Improves performance across multiple tasks;
• Preserves training stability.

The goal is not to reject as many updates as possible.

Rejecting too many candidates can prevent the model from exploring new reasoning paths. Accepting too many low-quality updates can introduce instability.

MIPU therefore aims to balance reliable improvement with continued exploration.

Experiments

The method was evaluated on Qwen3-1.7B and Qwen3-4B models under FP8 training settings.

The experiments covered several mathematical reasoning benchmarks, including:

• MATH-500;
• AIME24;
• OlympiadBench;
• Minerva;
• AMC23.

MIPU was compared with standard GRPO, MIS, and learning-rate decay.

Model	GRPO	MIS	Learning-rate decay	MIPU	Improvement over GRPO
Qwen3-4B	64.42%	63.42%	65.66%	66.71%	+2.29 points
Qwen3-1.7B	50.86%	51.73%	52.23%	53.97%	+3.11 points

MIPU achieved higher average accuracy on both model sizes.

On Qwen3-4B, MIPU reached 66.71% average accuracy, improving over standard GRPO by 2.29 percentage points.

On Qwen3-1.7B, MIPU reached 53.97%, improving over GRPO by 3.11 percentage points.

The method also showed more stable training trajectories, suggesting that the improvement was not limited to a single evaluation point.

Why candidate selection matters

The paper also evaluates the contribution of each stage:

Method	Average accuracy
Standard GRPO	64.42%
Step 1: candidate generation only	65.36%
Step 2: candidate evaluation only	62.81%
Full MIPU	66.71%

The results suggest that candidate generation and candidate evaluation play complementary roles.

Step 1 helps identify potentially useful updates. Step 2 helps determine whether those updates actually improve inference behavior.

Together, they allow MIPU to filter ineffective updates while preserving the model’s ability to explore new solutions.

Looking ahead

This work focuses on 1.7B and 4B models and mathematical reasoning tasks. Several directions remain open:

• Scaling MIPU to larger models;
• Applying the method to code generation and tool use;
• Evaluating it across different inference engines;
• Reducing the computational cost of candidate evaluation;
• Designing adaptive acceptance and rejection strategies;
• Extending the framework to more complex reward functions.

The broader message is simple:

Training optimization should ultimately be measured by the quality of the model’s real-world behavior.

MIPI and MIPU offer one possible direction: making the connection between training updates and inference performance more explicit, measurable, and stable.

Paper:

https://arxiv.org/abs/2606.29526

Project page:

https://anitaleungxx.github.io/MIPU