If you run reinforcement learning post-training on language models using frameworks like verl, TRL, or OpenRLHF, your rollout engine is probably serving weights that are dozens of optimizer steps out of date.
The issue is not network latency or worker contention. It comes from the cast that converts optimizer master weights into the weights served by your inference engine.
Modern post-training stacks keep master weights in fp32 inside PyTorch and serve rollouts from a bf16 copy hosted in vLLM or SGLang. Every open-source framework bridges those runtimes with standard round-to-nearest conversion.
A recent Stanford paper from Andrew Gao and William Gao (arXiv:2610.41073, code at github.com/andrewgcodes/faircast) shows that at standard RL learning rates, round-to-nearest quietly discards more than 99 percent of per-step parameter updates.
The bf16 dead zone
Bfloat16 has a 7-bit significand. For a weight of magnitude |w|, the distance between adjacent representable numbers is roughly |w| / 128.
Round-to-nearest maps any update smaller than half that distance straight to zero. That creates an arithmetic dead zone of |w| / 256.
When you train with Adam at typical post-training learning rates between 1e-6 and 3e-7, almost every per-parameter update falls below that threshold. On Qwen3-1.7B, only 0.4 to 0.5 percent of served weights actually change on any given optimizer step. A weight has to accumulate updates in the same direction for 50 to 200 consecutive steps before it crosses a bf16 grid boundary and changes the served model.
If you use Muon, whose per-element updates are roughly uniform in magnitude, the problem gets worse. At 1e-6, practically none of the updates clear the rounding threshold. The inference engine is generating rollouts from a model that is effectively frozen.
Why standard diagnostics miss it
Most RL stacks track training-inference mismatch through an importance ratio between the trainer forward pass and the sampler log-probabilities. Frameworks also use PPO ratio clipping, Truncated Importance Sampling, and IcePop token masking to guard against numerical drift.
None of those tools detect this lag.
In nearly every framework, the trainer forward pass runs in bf16 autocast or evaluates the exported bf16 weights directly. Both the trainer and the sampler evaluate weights downstream of the round-to-nearest cast.
Because both runtimes evaluate the same rounded copy, their visible ratio stays close to 1.0 throughout training. In the paper's measurements on MATH and logic tasks, the visible log ratio hovered around 0.01 per token.
The ratio that actually governs policy gradient updates is between the fp32 master weights and the served bf16 copy. Frameworks never form that ratio. While the visible metric looked clean, the hidden ratio grew to between 0.10 and 0.45, and effective sample size on 2,000-token responses dropped to between 0.2 and 0.35.
The saved checkpoint discrepancy
This rounding behavior also creates a gap between what you train and what you save.
At checkpoint time, frameworks save the fp32 master weights from the optimizer. On knights-and-knaves and zebra puzzles, the paper found that the saved fp32 checkpoint scored between 12 and 50 points lower than the bf16 model that actually generated the rollouts during the run.
If you take that saved fp32 checkpoint and convert it to fp16 or FP8 for production serving, your deployed model does not match the behavior of the model you evaluated during training.
The one-line fix: Fair Cast
The paper proposes Fair Cast, which fixes the problem with two changes:
- Replace round-to-nearest with stochastic rounding at the export barrier. When an fp32 weight falls between two bf16 numbers, round up with probability proportional to its fractional position. In expectation, the exported weights match the master weights exactly. Implementation requires adding 16 random bits to the fp32 representation before truncating. On a 1.7B parameter model, this takes 22 to 38 milliseconds.
- Evaluate training loss on the exact served copy, ensuring the differentiated policy matches the policy generating rollouts.
Across 25 matched seed pairs on Qwen3 models from 0.6B to 4B parameters, Fair Cast beat round-to-nearest on 18 runs, tied on 6, and trailed on 1. On zebra logic puzzles, where round-to-nearest collapsed to 0.33 accuracy, Fair Cast reached 0.57.
What to check in your stack
If you run RLVR, PPO, or GRPO on open-source frameworks, inspect the call that syncs weights from your PyTorch optimizer module to your inference workers.
If the sync uses a standard PyTorch cast like weight.to(torch.bfloat16), it is running round-to-nearest. At learning rates below 5e-6, your rollout engine is serving weights that lag your optimizer by dozens of steps.
Switching to stochastic rounding at the export call takes minimal code and adds negligible latency. If your reasoning runs plateau early while your loss curves look healthy, check your weight-export cast before tuning your rewards.
Top comments (5)
The hidden-ratio framing is what makes this land — I've been staring at PPO ratios stuck near 1.0 on runs that plateaued and never thought to check that both forward passes might be evaluating the same rounded copy. Grepping our verl sync path confirms it's a plain .to(torch.bfloat16), so round-to-nearest all the way down.
One thing I'm unclear on: with stochastic rounding you trade a zero-mean rounding error for a variance-increasing one, so at very small update magnitudes doesn't the served bf16 copy get noisier step to step even while being unbiased? 22–38ms at 1.7B sounds fine, but did the paper measure anything about rollout quality variance, not just the accuracy averages?
The paper tracks this directly in two ways.
First, at the parameter level, the rounding noise is zero-mean and independent across billions of weights, so it does not compound across a rollout. Under stochastic rounding, the per-token divergence |log(pi_theta / pi_served)| stays pinned near 0.01 (essentially the noise floor of the forward pass). Under round-to-nearest, because the 99% truncation is coherent in the same direction across 50 to 150 steps, the hidden drift compounded until effective sample size dropped to 0.20 to 0.35 on 2,000-token rollouts.
Second, on rollout behavior: round-to-nearest was accidentally acting as a low-pass filter, suppressing quick policy shifts. In Section 7.5, they tested a task (List functions) where removing that damping caused the policy to collapse response length early (it dropped from 600-token derivations to bare 13-character answers by step 29). Their fix was not reverting to round-to-nearest, but decoupling the smoothing: exporting an EMA of the master weights (alpha = 0.05, ~20 steps lag) with stochastic rounding kept full derivations and hit 0.708 accuracy versus 0.668 for round-to-nearest.
The important diagnostic is the distribution of update magnitudes relative to the representation threshold, not merely the final reward curve. If most updates quantize away, a run can look stable while learning has silently moved to a sparse, biased subset of parameters. I’d log the fraction of zeroed updates by layer and optimizer step so a precision change has a measurable causal story.
PPO ratios stuck near 1.0 is the symptom I'd actually recognise — I'd have blamed the training setup for weeks before suspecting the cast. The dead-zone math is brutal though: an update smaller than half the gap just vanishes. Is stochastic rounding something you'd actually run in production now, or is it still too noisy to trust on a real run?
The noise from stochastic rounding is zero-mean across billions of weights, so it does not act like random weight perturbation. The practical catch before running it in production is export latency. Adding 16 random bits on each export takes 20 to 40 milliseconds on a 1.7B model, but across multi-node clusters that adds up if you broadcast weights after every step. If your rollout sync can absorb that pause, it is much safer than letting 99 percent of updates sit in an arithmetic dead zone.