Multi-Reward RL, Part 2: Benchmarking GRPO, DAPO, and CISPO on Unseen Tasks

Follow-up: Part 3 scales the CISPO + REPO-R recipe to Qwen3.8-27B and 600 steps, with a one-change-per-run holdout ladder and an advantage-floor failure we found in a harsher environment.

Part 1 analyzed how PPO, GRPO, DAPO, and GDPO balance competing reward objectives in theory. In this follow-up empirical benchmark, we test the practical question engineering teams actually face: which trainer algorithm actually produces a model that solves brand-new, unseen problems?

We ran seven distinct RL algorithms on Qwen3-14B in our deterministic decentralized exchange (DEX) arbitrage gym across two reasoning regimes: No-think (direct execution) and Thinking-2048 (with chain-of-thought tokens enabled). The results revealed a severe cautionary tale for post-training teams.

The Generalization Trap: In thinking mode, CISPO achieved the highest training reward curve (0.8951 last-10 mean), yet its performance on frozen test tasks collapsed to 0.6068. Meanwhile, the raw Qwen3-14B base model—with thinking enabled and zero RL training—scored 0.6830. If you only look at your training curves, you will celebrate a run that is actively degrading your model.

Holdout native score on 10 unseen tasks for each trainer, without thinking (blue) and with 2,048 thinking tokens (orange). Without thinking every trainer beats the untrained model; with thinking only DAPO-refill does.

Start here

Understand why training reward curves often diverge from holdout generalization in multi-step reasoning, see how Qwen3-14B performs across GRPO, DAPO, GDPO, CISPO, ADAPO, and REPO-R, why an untrained thinking baseline is mandatory before declaring victory, and examine the compute costs of test-time reasoning.

• Holdout Generalization: Scoring the model checkpoint on completely frozen, unseen task parameters without updating weights on them.
• Surrogate Inversion: When a policy optimizer ruthlessly exploits the composite training reward formula but produces worse actual domain solutions.
• DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization): Token-level loss averaging with asymmetric clipping bounds (0.20 / 0.28).
• CISPO: Clipped Importance Sampling Policy Optimization that caps importance weights directly at 1.20 rather than clipping surrogate ratios.
• Thinking Baseline: Evaluating the raw base model with chain-of-thought tokens enabled, ensuring RL training adds genuine value beyond test-time compute.

1. What the agent does: The DEX gym

DEX stands for decentralized exchange. In our gym, the agent is presented with a frozen snapshot of automated market maker (AMM) liquidity pools: token reserves, exchange fees, gas prices, and routing constraints. The agent’s task is to find an optimal two-to-four pool arbitrage cycle and compute the exact input amount (amount_in_wei) that returns more of the starting asset after all fees and gas deductions.

WETH (Wrapped Ether) is the asset at both ends of the route. Consider an intuitive triangular arbitrage cycle: the model starts with WETH, swaps into USDC, swaps USDC into UNI, and swaps UNI back into WETH.

• If the trade size is too small: The fixed Ethereum gas fees eat 100% of the price discrepancy, yielding a negative net return.
• If the trade size is too large: The constant-product liquidity formula (x · y = k) causes severe price slippage, collapsing the margin.
• If the pool sequence is invalid: The simulated EVM contract reverts and the transaction fails immediately.

The agent interacts using structured tool calls: inspecting pool reserves, proposing a pool route and trade amount, evaluating feedback, and finalizing. The verifier replays the transaction bytecode using a native Foundry / revm EVM execution engine. It scores the verified mathematical execution on the simulated blockchain, rather than trusting any self-reported text output by the model.

Training reward = R_native + 0.05 × R_efficiency + 0.20 × R_reliability

The composite training reward consists of three distinct channels:

• Native Task Score (R_native ∈ [0, 1]): The verified net profit margin returned to the starting wallet after all pool fees and gas costs.
• Execution Efficiency (R_efficiency ∈ [0, 1]): Rewards solving the task compactly in 2–3 tool turns instead of exhausting all 6 turns: max(0, 1 − tool_steps / 6).
• Reliability Gate (R_reliability ∈ {-1, +1}): +1.0 if the transaction executed cleanly without syntax errors or contract reverts; −1.0 if the call failed.

The training reward adds efficiency and reliability bonuses to the native task score, so it can move without the task score moving. Holdout results report the native score alone.

2. What we held fixed: The training contract

To isolate the impact of the RL trainer algorithms, every other variable was strictly controlled:

• Model and Quantization: Qwen3-14B (revision 40c06982…), quantized NF4 weights with BF16 compute, LoRA rank 32, alpha 64.
• Compute Hardware: One dedicated NVIDIA RTX PRO 6000 Blackwell 96 GB per training run, Hugging Face generation, fused SDPA attention, CUDA graphs disabled.
• Optimizer Schedule: Fixed seed 42, learning rate 1e-5 with linear decay, 50 optimizer steps, group size 4, eight completions per fresh rollout batch, two policy updates per rollout. Curriculum learning disabled.
• No-think mode: Chain-of-thought thinking disabled; max 1,024 generated tokens per tool turn (max 8,192 trajectory cap).
• Thinking-2048 mode: Up to 2,048 thinking tokens inside a 3,072-token turn budget (max 22,528 trajectory cap).
• Frozen Holdout Evaluation: Evaluated on the final step-50 checkpoint. 5 task families × 2 held-out seeds × 4 samples = 40 episodes across 10 distinct unseen scenarios. Untrained starting checkpoints evaluated identically in both reasoning modes.

3. Trainer features: Seven algorithm bundles

In modern post-training, an RL algorithm is not a monolithic block. It is a bundle of three distinct modular decisions: which rollouts enter the training batch, how multiple rewards are combined into policy advantages, and how that advantage scales policy gradient updates.

Configuration	Loss / averaging	Reward → advantage	Dynamic sampling	Control
GRPO	Clipped surrogate; sequence-level averaging	Joint reward → group normalization	Off	Monitor only
DAPO	Clipped surrogate; token-level averaging	Joint reward → group normalization	Off	Fixed asymmetric clip: 0.20 / 0.28
DAPO-refill	Same DAPO loss	Same as DAPO	On; at most 2 refill rounds	Same fixed clip
GDPO	DAPO loss in this implementation	Normalize each reward, then combine	Off	Fixed asymmetric clip
CISPO	Detached, capped importance weights	Joint reward → group normalization	Off	Importance-weight cap 1.20
DAPO + ADAPO	DAPO loss	Same as DAPO	Off	Entropy feedback adjusts upper clip
DAPO + REPO-R	DAPO loss	Token-level advantage shaping	Off	Entropy feedback controls ζ
CISPO + REPO-R w5	CISPO loss; cap stays 1.20	Token-level advantage shaping	Off / separate refill arm	5-step target window; ζ ≥ 0

GDPO decouples reward-channel normalization before combining advantages. CISPO replaces ratio clipping with direct importance weight clipping (capped at 1.20). ADAPO and REPO-R introduce adaptive entropy control: ADAPO adjusts clipping bounds based on policy entropy, while REPO-R shapes token-level advantages. DAPO-refill activates replacement sampling when a rollout group exhibits zero reward variance, preventing zero-gradient wasted batches.

4. Full results: Train vs. holdout test

The CISPO Inversion Trap

CISPO dominated training reward curves but collapsed on held-out tasks. One likely reading: it kept raising the composite training reward without improving the native score on tasks it had not seen.

Thinking Baseline Beats RL

Untrained Qwen3-14B with 2,048 thinking tokens beat 6 of the 7 trained models. Always benchmark against an untrained thinking baseline.

DAPO Leads Generalization

DAPO delivered the highest native holdout score in no-think mode (0.6467, 39/40), and DAPO-refill led thinking mode (0.7105, 38/40).

The 5× Compute Tax

Thinking steps took 4.5–5.9× longer wall-clock time. You must balance the evaluation gain (+0.0275) against a 5× compute budget increase.

Thinking mode: training reward over the last ten points against holdout score. CISPO had the highest training reward and the lowest holdout score.

No-think mode: Every trained model beats the starting baseline

Configuration	Steps	Train mean	Train last-10	Holdout native	Successes	Step, s	Step-hours
Starting model (baseline)	0	—	—	0.3856	16/40	—	—
GRPO	50/50	0.7038	0.7729	0.6424	38/40	177	2.46
DAPO (top holdout)	50/50	0.6519	0.7142	0.6467	39/40	176	2.45
DAPO-refill	50/50	0.6564	0.7585	0.5839	34/40	224	3.11
GDPO	50/50	0.6488	0.7727	0.6245	36/40	167	2.32
CISPO	50/50	0.6486	0.7486	0.6190	36/40	164	2.27
DAPO + ADAPO	50/50	0.6694	0.7525	0.6332	37/40	168	2.33
DAPO + REPO-R	50/50	0.6373	0.7057	0.5724	31/40	156	2.16

Without reasoning tokens, post-training delivers an unambiguous leap forward. The untrained base model scored only 0.3856 (passing 16/40 episodes). DAPO achieved 0.6467 (39/40 successes), closely followed by GRPO at 0.6424. Every single trained configuration substantially outperformed the starting model.

Thinking-2048 mode: The starting model sets a high bar

Configuration	Steps	Train mean	Train last-10	Holdout native	Successes	Step, s	Step-hours
Starting model (baseline)	0	—	—	0.6830	37/40	—	—
GRPO	50/50	0.8362	0.8824	0.6798	38/40	929	12.90
DAPO	50/50	0.8036	0.7916	0.6703	38/40	938	13.02
DAPO-refill (top holdout)	50/50	0.8275	0.8564	0.7105	38/40	1016	14.10
GDPO	50/50	0.8190	0.8253	0.6602	35/40	938	13.03
CISPO (diverged)	50/50	0.8388	0.8951	0.6068	33/40	907	12.60
DAPO + ADAPO	50/50	0.8185	0.8815	0.6250	34/40	887	12.32
DAPO + REPO-R	50/50	0.8285	0.8206	0.6461	34/40	922	12.81
CISPO + REPO-R w5	50/50	0.8165	0.8332	0.6243	34/40	937	13.01
CISPO + REPO-R w5 + refill (partial)	41/50	0.7908	0.8302	—	—	912	10.39

When chain-of-thought thinking tokens are enabled, the starting model achieves 0.6830 straight out of the box (passing 37/40 episodes). Only DAPO-refill surpassed the untrained model on native holdout reward (0.7105, an incremental gain of +0.0275). GRPO (0.6798) and standard DAPO (0.6703) finished slightly below the base model’s score despite high success rates (38/40), demonstrating that success counts and solution quality answer different questions.

5. Explore the curves: Rollout telemetry

Each 50-step run generated 25 fresh rollout reward checkpoints (even optimizer steps update on rollouts generated on odd steps). The chart below shows three of them in thinking mode:

Raw training reward in thinking mode for CISPO, DAPO and DAPO-refill. One batch of eight completions per point, so the curves are noisy; the late averages and the holdout scores tell different stories.

Over its last ten reward points CISPO averaged 0.895 in thinking mode and DAPO 0.792, yet on unseen tasks DAPO scored 0.670 and CISPO 0.607. The training reward is measured on training tasks and includes bonuses, so it is no substitute for a frozen holdout.

6. Time and memory: The cost of thinking

Test-time reasoning comes with a heavy computational invoice. Across all seven algorithms, an average thinking step took 4.5–5.9× longer than a no-think step. Fifty training steps required approximately 12.3–14.1 hours of GPU execution in thinking mode, compared to 2.2–3.1 hours in no-think mode.

Mean seconds per optimizer step. Thinking made steps 4.5 to 5.9 times slower, for at most +0.028 over the untrained model on holdout.

Configuration	Peak allocated, GiB	Peak reserved, GiB
GRPO	69.1	76.9
DAPO	66.5	74.0
DAPO-refill	68.5	76.3
GDPO	68.3	76.1
CISPO	70.0	77.8
DAPO + ADAPO	71.8	79.6
DAPO + REPO-R	68.0	75.5
CISPO + REPO-R w5	66.9	74.4
CISPO + REPO-R w5 + refill (partial)	70.3	77.8

The table covers the thinking-mode runs: peak memory stayed between 66.5 and 71.8 GiB allocated (74.0–79.6 GiB reserved), within the 96 GB budget of the Blackwell GPU. The primary operational bottleneck is wall-clock rollout latency, not GPU memory.

7. Combining features: The CISPO + REPO-R hybrid

Can we rescue CISPO’s optimization speed while fixing its generalization collapse? We tested a hybrid architecture: using CISPO’s capped importance weights (cap 1.20) combined with REPO-R’s token-level advantage shaping.

To stabilize entropy dynamics, we introduced a nonnegative window-5 controller: the first 5 optimizer steps record entropy drift without intervention (ζ = 0). Once calibrated, the entropy control strength is constrained strictly to 0 ≤ ζ ≤ 0.05.

The completed hybrid achieved 0.6243 native holdout reward and 34/40 successes, outperforming standalone CISPO (0.6068, 33/40). While this +0.0175 lift confirms that advantage shaping mitigates policy collapse, the hybrid still trailed the raw untrained thinking baseline (0.6830).

Enabled is not the same as exercised

In no-think mode, DAPO-refill triggered 19 replacement rollout rounds. In thinking mode, DAPO-refill triggered zero refills—the model never produced an all-identical reward batch. Therefore, we cannot attribute the thinking DAPO-refill lead (0.7105) to the refill mechanism itself; it functioned identically to standard DAPO under that seed.

8. What to test next: Next iteration & replication

Based on these findings, our production post-training recommendations for multi-step reasoning workflows are:

1. In No-Think Mode: Standard DAPO and GRPO are clear, cost-effective winners. They raised the holdout score from 0.386 to about 0.64–0.65 (16/40 to 38–39/40 successes) in about 2.5 GPU-hours.
2. In Reasoning Mode: Always evaluate your untrained base model with reasoning enabled before declaring training success. DAPO-refill is the primary candidate for replication.
3. Never Pick a Checkpoint by Training Curves: CISPO proved that soaring training curves can mask test-set degradation. All promotion decisions must be gated by frozen holdout benchmarks.
4. Multi-Seed Replication: Expand seed sweeps (seeds 43, 44) across novel scenario families to confirm that DAPO-refill’s +0.0275 edge holds across market distributions.

Evidence and method references

All raw empirical telemetry is publicly available. The public JSON export contains complete run manifests, SHA256 hashes, all 791 recorded optimizer steps, 396 fresh reward logs, and 680 evaluated holdout episodes. The Markdown companion provides searchable per-step records.

Foundational algorithm publications: DeepSeekMath / GRPO; DAPO; GDPO; MiniMax-M1 / CISPO; ADAPO and REPO-R.

---

Applying this to your own models? We build private RL training environments with deterministic verifiers for teams' own workflows. Private training environments.

Originally published at g-ftech.com.