Distillation has been treated as an imitation shortcut, but TREK shows it can power the first step of exploration on hard prompts. By pulling verified solution modes into a student model before any reinforcement‑learning fine‑tuning, it sidesteps the instability that typically plagues RL‑only pipelines.
Before TREK, most attempts to broaden a language model’s reasoning space relied on on‑policy reinforcement methods such as Group Relative Policy Optimization (GRPO). Those approaches assume the current policy already samples useful trajectories and stall when faced with prompts whose correct answers lie outside the model’s support. Consequently, developers have been forced to endure long RL runs just to reach marginal gains.
TREK expands the student’s solution support by distilling verified proposals through a short forward‑KL phase. “TREK first identifies prompts where the unaided student has very low pass rate, queries a proposal source to produce verified candidate solutions, keeps the top‑r proposals ranked by current student likelihood, applies a short forward‑KL phase to pull those verified modes into the student’s support, and then returns to standard on-policy GRPO refinement.” The forward direction “penalizes missing proposal support and encourages the student to cover the retained solution modes,” directly replacing RL for the exploration step [1].
On challenging mathematical reasoning benchmarks, TREK lifts Qwen3‑8B’s AIME 2025 score from 36.9% to 40.3% (average over 16 samples) while also raising its AIME 2024 performance from 47.9% to 51.1% after a single forward‑KL pass with DeepSeek‑V4 proposals [1].
When the same model generates proposals using self‑context instead of an external teacher, it still gains noticeable jumps: the AIME 2025 score reaches 38.5% and AIME 2024 climbs to 49.6%, demonstrating that even black‑box teachers are unnecessary for most gains [1].
Agentic tasks benefit similarly; ScienceWorld success surges from 12.5% to 26.7%, and ALFWorld accuracy improves from 75.8% to 82.8% after TREK’s exploration stage, with the hardest task types solving early in training compared to vanilla GRPO [1].
The method adds teacher queries and verifier calls on top of ordinary GRPO, so computational overhead is non‑trivial; “more adaptive scheduling could reduce this overhead at the cost of additional engineering.” Moreover, TREK still relies on a subsequent RL refinement phase to polish policies, leaving open how far pure distillation can go without any reinforcement signal.
If these gains hold across downstream tasks, teams can replace the costly initial RL exploration loop with a single forward‑KL distillation stage, rerunning benchmarks like AIME or ALFWorld under the TREK pipeline before committing resources to on‑policy optimization.
Top comments (0)