The mismatch inside standard on-policy distillation
On-policy distillation trains a student on responses sampled from the student’s current policy, then asks a stronger teacher to provide token-level guidance on those trajectories. In the formulation used by the GC-OPD paper, each sampled token receives an advantage based on the difference between the teacher’s and student’s log-probabilities.
This signal is dense and directly attached to the tokens the student generated. But it answers a narrow question: does the teacher prefer this token over the student’s current policy?
That is not necessarily the same as asking whether the complete response satisfies the task.
The distinction matters when an answer must combine evidence scattered throughout a large input. A response can remain locally plausible while omitting required evidence or violating a global constraint. The teacher may still assign favorable token likelihoods because many individual continuations look reasonable. A task-specific verifier instead scores the completed response, potentially using graded rewards to represent partial success.
This concern is consistent with the broader observation that nominal context capacity does not guarantee reliable integration of distant information. LongBench v2 focuses on understanding and reasoning across complex long inputs, while LongBench Pro distinguishes tasks requiring global integration from those depending on localized retrieval. GC-OPD focuses specifically on the resulting post-training problem: dense teacher preference and response-level task completion can disagree.
Diagnosing teacher–verifier disagreement
For each input, standard OPD samples a group of student responses. A response’s trajectory-level OPD score is the mean of its token advantages:
s = mean(token OPD advantages)
A task-specific verifier independently assigns the response a reward, R.
The paper analyzes fixed responses from two representative evidence-aggregation tasks using pairwise disagreement rate and OPD preference gap. It reports that trajectory-level OPD scores become progressively less aligned with verifier rewards across longer prompt-length ranges.
This diagnosis does not imply that the teacher signal is generally useless. Instead, it identifies a granularity mismatch:
- The teacher supplies dense, token-level preferences.
- The verifier measures task completion at the response level.
- Standard OPD has no explicit mechanism for correcting cases where those assessments conflict.
Replacing the teacher signal entirely with response rewards would discard its dense supervision. GC-OPD instead treats verifier feedback as a calibration term.
How GC-OPD constructs a signed residual
Group-relative assessments
GC-OPD first evaluates every response relative to the other rollouts generated for the same input. Within each rollout group, it separately applies z-score normalization to verifier rewards and trajectory-level OPD scores:
normalized_reward = z(R)
normalized_OPD_score = z(s)
This makes the two signals comparable as within-group relative assessments rather than as raw values with potentially different scales.
The implementation uses group-level and token-level standard-deviation thresholds of 10^-6 for numerical stability.
The disagreement residual
For each response, GC-OPD computes:
residual = normalized_reward - normalized_OPD_score
The sign carries the key information.
A positive residual means the response ranks better under the verifier than under the teacher-derived trajectory score. The training update should therefore give it more credit than vanilla OPD would.
A negative residual means the teacher-derived score is relatively more favorable than the verified outcome. The update should reduce that endorsement.
Because this is a difference rather than a direct reward addition, calibration concentrates on disagreement. When the two assessments rank a response similarly, the residual is small even if both normalized values are individually large.
RACA turns response feedback into token credit
The residual belongs to an entire response, but optimization still operates over tokens. Assigning the same residual adjustment to every token is possible, yet it ignores the structure already present in the teacher’s dense signal.
Relative-advantage-based credit assignment, or RACA, starts by normalizing each token’s OPD advantage relative to the response mean. It then maps the result to positive, bounded credit:
credit_t = 1 + tanh(relative_advantage_t / 2)
The final token advantage is:
adjusted_advantage_t = OPD_advantage_t + beta × credit_t × residual
The experiments use a residual coefficient beta of 0.10.
RACA therefore does not replace the original OPD advantage. It preserves that signal and uses its relative pattern to determine how strongly each token receives the response-level correction. The method requires neither step labels nor auxiliary continuations.
What the paper tested
The authors trained Qwen3-4B and Qwen3-8B students using Qwen3-30B-A3B-Thinking as the teacher. Training used 9,527 prompts from GoLongRL after filtering inputs at 32K tokens, with a shared horizon of 100 training steps.
The data covered nine task families and both binary and graded verifier interfaces. The largest groups were precise long-range retrieval with 4,693 prompts and evidence-grounded reasoning with 3,204. Prompt lengths ranged from at most 2K to 24–32K tokens.
Across five reported tasks, the aggregate results were:
| Student | Raw | Vanilla OPD | GC-OPD |
|---|---|---|---|
| Qwen3-4B | 29.08 | 39.31 | 40.47 |
| Qwen3-8B | 35.12 | 43.56 | 44.65 |
These are paper-reported results, not independent replications. The clearest interpretation is incremental: most of the aggregate improvement over the raw checkpoints came from OPD, while group-relative calibration added 1.16 points for the 4B model and 1.09 points for the 8B model over vanilla OPD.
The gains also varied by task. For Qwen3-8B, GC-OPD improved CorpusQA from 39.82 under OPD to 43.77, but LBv1QA moved from 57.80 to 58.30 and Frames remained at 34.59.
What the ablations tell developers
On Qwen3-8B, adding another OPD-derived response term improved the aggregate result by only 0.04 over vanilla OPD. Directly adding group-normalized verifier reward improved it by 0.63. The signed disagreement residual improved it by 1.10.
Credit allocation also mattered. Absolute-OPD allocation added 0.38, uniform allocation added 0.72, and RACA added 1.10 over vanilla OPD.
For post-training systems, the practical lesson is not simply “add verifier rewards.” Keep teacher and verifier assessments separate, normalize them among responses to the same input, and use their signed difference to correct only the mismatch. Then allocate that correction without erasing the original token-level guidance.
Limitations and practical scope
The evidence comes from two Qwen3 student sizes, one teacher, one filtered training collection, five reported tasks, and a 100-step training setup. Results therefore do not establish how the method behaves with other model families, teachers, group configurations, or verifier quality levels.
GC-OPD also depends on variation within rollout groups. Its normalization requires explicit stability thresholds, and its correction is only as task-relevant as the verifier rewards being compared.
Within that scope, GC-OPD offers a focused modification for developers who already have student rollouts, teacher log-probabilities, and response-level verifiers. It preserves dense distillation while making verified task success an explicit, signed correction rather than a competing objective.
Tags: ai, machinelearning, llm, opensource
Top comments (0)