If you're evaluating verifiable-reward RL (RLVR / GRPO / R1-style outcome-only training) with greedy accuracy alone, this post is about a failure mode you cannot see: the metric can say "improving" while the search-augmented capability your deployment depends on is quietly being destroyed. We built a fully observable toy system to test when outcome-only RL creates competence vs. merely reallocates search — the answer is a six-regime taxonomy, and two of its regimes are evaluation blindspots with concrete diagnostics.
The setup
A 1.8M-parameter transformer trained on three synthetic algorithmic families (counting, multi-digit addition with a carry class, digit-sum parity) with exact ground truth. The key trick: instead of treating base competence as a fixed property of a large pretrained model, we swept it as a controlled variable (p0 = the base's greedy accuracy on the failing class, from 0 to 0.98), then compared outcome-only GRPO against a matched-budget base+search baseline (pass@k sampling from the frozen base) at the same seed, same eval prompts (held-out seed 777, decoupled from training seed).
Why this matters: recent analyses (BOPTR and others) show much of RLVR's apparent gain over the base is recoverable by test-time search at matched budget — RLVR as budget transfer, not capability creation. But those analyses never manipulate base competence, so the regime where RLVR genuinely creates — versus merely reallocates — is never swept. That's the gap this study fills.
Six outcomes, not one
The result is not a threshold. It's a regime taxonomy:
| Regime | Base condition | What happens |
|---|---|---|
| WALL | p0 ≈ 0 (zero sampling support) | RL and search both 0.000 at every feasible budget — ~60k rollouts produced zero correct samples (per-sample p < 1.2e-4 at 95% CI). All-fail GRPO groups carry zero advantage → zero gradient. Nothing to bootstrap. |
| RACE | p0 ≈ 3e-3 (rare bootstrap) | A seed lottery: 1/3 of seeds reproduce an 8.3×-over-search "money cell" (RL pass@64 0.688 vs base+search 0.083), 1/3 weak, 1/3 stuck at the wall. Held-out eval seed reproduces the exact rank order — the lottery, not the draw, is the reproducible claim. |
| CREATE | p0 0.03–0.13, representable rule | RL is robustly load-bearing (3/3 seeds): lifts each seed's greedy 5–15× over its own SFT base, compresses the base's own seed-lottery (0.034 → 0.51), expands per-prompt sampling entropy (0.81 → 1.22 bits). This is the regime most small-model RLVR success stories operate in — and it's one of six. |
| DESTROY | p0 > 0 but wide answer space (99 values) | RL contracts per-prompt completion entropy 4× (2.19 → 0.55 bits). Greedy dips then recovers to base by 1,500 steps — but pass@64 collapses budget-monotonically (0.83 → 0.19) and never recovers. The sampling channel that base+search relied on is gone. |
| GREEDY-BLIND | Imbalanced SFT (90/10 parity) collapses the argmax | Odd-class greedy stays 0.000 through ~3,840 odd examples in every RL run while base sampling mass grows to pass@64 = 1.0. Whether RL preserves that sampling channel is itself unstable across runs: post-RL pass@64 ≈ 1.0 in 2/3, ≈ 0.02 in 1/3 — in a 2-value answer space. Balanced SFT learns the rule perfectly, so this is imbalance collapse, not unlearnability. |
| NO-GAP | p0 ≈ 0.98 | RL sustains, adds nothing. |
Registered priors were part of the design: the amendment P1′ (load-bearing RL appears iff base failures are systematic) is refuted in its strong form — systematic near-zero-p0 failure is a wall for both search and RL. The two conditions that actually matter: bootstrap support (p0 > 0 at the group scale) AND reinforceable partial structure (rewards transfer across a prompt-equivalence class big enough to learn from). p0 alone doesn't determine the outcome — at matched p0 ≈ 0.15, count is load-bearing 3/3 while add degrades 3/3.
The blindspot, in one graph
The DESTROY budget trajectory is the one to internalize:
- greedy (carry): dips at 500 steps, recovers to base by 1,500 steps
- pass@64 (carry): 0.833 → 0.188, monotonically, never recovers
A greedy-only evaluator would declare the system "recovered" at step 1,500. The search capability is gone. The converse blindspot is GREEDY-BLIND: a greedy-only evaluator reports "RLVR did nothing" (odd greedy 0.000 before and after) while sampling already carried the answer to pass@64 = 1.0 — and in 1/3 of independent runs RL destroyed even that.
The mechanism is measured, not speculated: outcome-only RL raises per-sample correct probability ~3× in both families (it's not inert, not memorizing — seen-vs-fresh operand pairs falsify memorization). The divergence is entirely in what happens to sampling support: count's learned rule populates the full correct answer set (entropy up, pass@k up); add's partial rule peaks onto a narrow band of sums (87.5% of its correct carry answers sit at sums 110–119 — an island of width ~1.7 in a 99-value space), so RL sharpens onto the island and pass@64 falls even as per-sample p triples. A KL anchor (beta 0.01) neither causes nor prevents either direction (ablation verified).
What this means for your eval stack
- Greedy accuracy is not a sufficient RLVR monitor. Add a sampling-based readout (pass@k / coverage of correct answers) at fixed prompts before and after training. The paper reports greedy-up/pass@k-down and greedy-flat/pass@k-strong as two checkable signatures.
- Check what your base's competence looks like. Diffuse partial structure (a few correct answers scattered in a wide space, or an argmax collapsed by training imbalance) is exactly the profile that gets destroyed or never created. Test-time search on such a base is not a safe fallback after RL — it may have been the only thing working.
- "RLVR does nothing" and "RLVR works" are both possibly wrong under greedy-only eval. Report both channels.
Study provenance
This is a published manuscript with pre-registered priors, six data figures, a numbered table, statistical-precision appendix, and an end-to-end reproduction harness (reproduce.sh, pinned seeds, all checkpoints). Full write-up: github.com/argszero/silicon-science-cs/tree/main/papers/issue-79.
Disclosure: this study was published in an open peer-reviewed journal I help operate (SILICON SCIENCE · Computer Science) — the manuscript, data, and code are public; the editor ran an independent clean-clone replication of the seed-lottery cells before acceptance.
Top comments (0)