DEV Community: Aleksei Romanov

GPU Kernels, Part 2: Teaching a 27B Model to Write Triton Kernels with SFT and a GPU Gym

Aleksei Romanov — Tue, 03 Nov 2026 18:11:19 +0000

In Part 1, we explored how GPU kernels work under the hood — why operator fusion and memory tiling matter, and how benchmarks like KernelBench evaluate whether an LLM can write kernels that actually beat PyTorch. This post is the hands-on sequel. We took an open-weight model, Qwen3.8-27B, and trained it to write Triton kernels across two stages: supervised fine-tuning (SFT) on 2,034 high-quality kernels re-verified on an NVIDIA H100, followed by reinforcement learning (RL) inside an automated GPU gym that compiles, validates, and benchmarks every candidate.

Evaluated on the standard 200 KernelBench problems (using a single greedy pass per problem), every headline metric moved in the right direction. Overall correctness improved from 21.5% to 29.5%. Kernels that were both mathematically correct and faster than eager PyTorch grew from 7.0% to 9.5%. On Level 1 (isolated operators), the model now solves 47 problems instead of 31 , and beats PyTorch on 15 instead of 8 .

At the same time, frontier commercial models still hold a substantial lead. Below, we break down how we built the dataset and verification gym, what each stage of training achieved, the subtle reward-design pitfalls where the gym taught the model the wrong strategy, and what it will take to close the remaining performance gap.

Open weights and data: the released checkpoint is available on Hugging Face as Qwen3.8-27B-GFT-Triton-Kernels-LoRA, complete with a vLLM-ready configuration and our exact evaluation harness. The SFT training set is available as GFT-Triton-Kernels-SFT-H100-Verified: 2,034 PyTorch-to-Triton pairs, each compiled and verified on an H100. Every metric cited in this post is included in the JSON download.

KernelBench Triton, levels 1–2 (200 problems), one greedy answer per problem, H100, FP32. Base model, after SFT, and after SFT + RL (the released checkpoint).

Metric	Base	After SFT	After SFT + RL	Change
Answers that compiled and ran	79.5%	86.5%	88.5%	+9.0 pp
Correct (fast₀)	21.5%	26.0%	29.5%	+8.0 pp
Correct and faster than eager PyTorch (fast₁)	7.0%	8.0%	9.5%	+2.5 pp
… more than 1.5× faster (fast₁.₅)	5.0%	5.0%	6.0%	+1.0 pp
… more than 2× faster (fast₂)	3.5%	4.5%	5.0%	+1.5 pp
Correct and faster than torch.compile	12.5%	15.0%	16.0%	+3.5 pp
Level 1 correct	31.0%	43.0%	47.0%	+16.0 pp
Level 1 faster than eager	8.0%	13.0%	15.0%	+7.0 pp
Level 2 correct	12.0%	9.0%	12.0%	±0.0 pp
Level 2 faster than eager	6.0%	3.0%	4.0%	−2.0 pp

Start here

How we built an SFT set of kernels verified on the target GPU; how the kernel gym grades an answer and turns it into a reward; seven gym versions and the failure behind each one; what SFT and RL each changed on KernelBench; why level 2 did not improve; how frontier models compare on the same 200 problems.

SFT: Supervised fine-tuning: training on example prompts with verified target answers.
RL / GRPO: Reinforcement learning: the model generates 8 answers per task; candidates scoring above group average receive positive policy gradient.
Gym: The execution environment serving tasks and grading answers: a reference PyTorch module and an isolated H100 worker that benchmarks the result.
Verifier gate: A binary pass/fail check. Failing a hard gate zeroes the episode reward regardless of other qualities.
Hybrid answer: A solution keeping compute-heavy primitives in PyTorch (cuDNN convolution or cuBLAS GEMM) while writing Triton for the fused activation tail.
fast₁: Share of all problems where the answer is mathematically correct and strictly faster than the baseline. fast₀ is the correctness rate.
LoRA: Low-Rank Adaptation: updating small rank-decomposition matrices instead of all 27B weights. Our adapter uses rank 8.

1. The scoreboard

To keep comparisons rigorous and fair, every model in this post was evaluated against the exact same test harness: KernelBench (levels 1 and 2, Triton backend) using byte-identical prompts and the same verification runner on an NVIDIA H100 SXM. Level 1 covers 100 standalone operators (matrix multiplications, convolutions, normalizations, activations, and loss functions). Level 2 tests 100 composite modules — typically a heavy GEMM or convolution followed by an elementwise tail where fusion offers clear headroom. (Part 1 details how KernelBench evaluates submissions and the key caveats when comparing benchmarks).

Evaluation protocol for every Qwen checkpoint in this post.


Benchmark	KernelBench at revision `423217d`, Triton backend, levels 1–2, 200 problems
Prompt	KernelBench’s one-shot Triton prompt for FP32, system prompt “You are a helpful assistant”
Decoding	vLLM, greedy (temperature 0), thinking off, at most 8,192 new tokens, one answer per problem
Correctness	5 random inputs, FP32, `atol = rtol = 1e-4`
Timing	Mean of 100 CUDA-event runs against eager PyTorch and `torch.compile(mode="max-autotune")` on the same GPU
Hardware	NVIDIA H100 80GB HBM3; PyTorch 2.9.1, Triton 3.5.1, CUDA 13.0

The two training stages and the benchmark. Each stage starts from the previous adapter; the numbers under each box are KernelBench scores of that checkpoint on all 200 problems.

2. What SFT and RL changed

Out of the box, base Qwen3.8-27B already understands basic Triton syntax — roughly four out of five generated answers compile and execute without syntax errors. What it lacks is numerical precision (getting boundary conditions and indexing math right) and the architectural intuition needed to beat PyTorch. Our two-stage training addressed both:

Base model (grey), after SFT (light blue) and after SFT + RL (blue) on the same 200 problems. Every overall metric improved; level 1 improved most. Level 2 correctness did not move.

Higher compilation reliability: Solutions that compile and execute cleanly rose from 79.5% to 88.5% . On Level 2, the improvement was even sharper ( 76 → 94 ), as the model learned to avoid invalid tensor shapes and illegal memory accesses that trigger driver crashes.
Correctness increased by over a third: Passing all verification checks went from 43 to 59 out of 200 tasks. On Level 1, the jump was especially pronounced ( 31 → 47 ), representing a +50% gain in verified standalone operators.
Speedup followed accuracy: Kernels that were both mathematically correct and faster than eager PyTorch climbed from 14 to 19 tasks. Against torch.compile, fast solutions grew from 25 to 32 , while 2×+ speedups rose from 7 to 10 . On Level 1, the count of fast kernels nearly doubled ( 8 → 15 ).

Supervised fine-tuning provided the foundation for correctness (Level 1: 31 → 43 ), while RL further refined edge-case handling. Because every checkpoint was evaluated on the exact same problem suite, we can perform a paired bootstrap analysis. Compared to the SFT baseline, the final RL checkpoint successfully resolved 8 previously failing problems while regressing on only 1 :

Change against the SFT adapter on the same 200 problems, in percentage points, with a 95% paired bootstrap interval (100,000 resamples of problems). Four checkpoints of one 300-step RL run.

RL checkpoint	Correct	Faster than eager	Faster than torch.compile
Step 100 (released)	+3.5 [+1.0; +6.5]	+1.5 [−0.5; +4.0]	+1.0 [±0.0; +2.5]
Step 150	+3.0 [±0.0; +6.0]	+1.5 [±0.0; +3.5]	±0.0 [−2.0; +2.0]
Step 200	+1.0 [−1.5; +3.5]	+0.5 [−1.0; +2.5]	+0.5 [−1.0; +2.5]
Step 300	+1.5 [−2.0; +5.0]	±0.0 [−2.0; +2.0]	−1.0 [−3.0; +1.0]

The bootstrap confidence interval confirms that step 100’s correctness gain is statistically meaningful: a + 3.5 percentage point lift, with a 95% interval of [ +1.0 , +6.5 ]. Speed gains over SFT were more modest, with confidence bounds grazing zero. Beyond step 100, performance plateaued; because we selected step 100 with visibility into later checkpoints, its numbers reflect a best-of-run selection. Level 2 remained the notable outlier: overall correctness held at 12 problems while fast kernels slipped slightly from 6 to 4. Section 6 unpacks the mechanism behind this plateau.

3. Stage 1: an SFT set verified on the target GPU

Public web-scraped kernel datasets look substantial on paper, but they suffer from a major qualification issue: most examples were validated only on tiny toy inputs, run on consumer GPUs with loose tolerances, or never executed at all. For SFT to teach realistic GPU behavior, we built a filtered dataset where every single sample had to pass rigorous execution checks on an NVIDIA H100 at production scale:

How 6,875 public answers became 1,352 training rows, plus 682 successful kernels from our earlier gym runs.

Curated open source: We started with Kimi K2 solutions for KernelBook modules from the siro1 collection. Each problem was refactored into KernelBench’s standardized interface (Model → ModelNew, positional get_init_inputs), capping inclusion at two solutions per module.
Realistic tensor scales: KernelBook reference shapes often default to toy dimensions like 4 × 4 × 4 × 4, where execution time is dominated by driver overhead rather than memory bandwidth. We scaled tensor dimensions up to ~1,000,000 elements and re-verified all kernels under these production workloads.
Rigorous decontamination: Any problem with token-set Jaccard similarity exceeding 0.6 against any of the 270 KernelBench problems was purged.
Pure Triton enforcement: Static AST checks stripped out solutions that cheated by falling back to PyTorch operators inside forward(). While intended as a quality bar, Section 6 explains how this filter later backfired on composite modules.
Gym replay data: We added 682 verified successful solutions from earlier gym exploration runs (capped at 40 per operator family to prevent simple elementwise tasks from overwhelming the distribution).

The final dataset comprises 2,034 verified pairs — 1,624 of which outperform torch.compile at target scale — all within a 4,096 token context window. We trained a rank-8 LoRA adapter for 2 epochs (3,674 steps, 2e-4 learning rate) on a single H100 in roughly five hours.

4. Stage 2: the kernel gym

While SFT demonstrates what well-structured kernels look like, reinforcement learning gives the model agency to experiment, debug compiler errors, and receive immediate reward signals. Our kernel gym follows an OpenEnv-style architecture: in each episode, a worker serves a reference PyTorch module, the model drafts an implementation, and an isolated GPU subprocess benchmarks the candidate on an H100.

One episode. The task is a PyTorch module; the model answers once or repairs with tools; five verifier gates decide what the answer earns; the speed share of the reward grows logarithmically from 0.75× to 4×.

Tasks

The gym generates tasks across fourteen distinct families matching the operational footprint of KernelBench levels 1 and 2. Each task is dynamically instantiated from a procedural seed specifying the mathematical operator, tensor shapes (scaled from 1 to 32 million elements to ensure execution time reflects memory bandwidth rather than kernel launch jitter), and test distributions. For difficult operations where the model initially had near-zero success, we introduced staged curriculum variants: a small shape, a scaffold providing host setup and tile offsets with the inner math left blank, and an epilogue task where the core library GEMM remains intact while the fused activation tail is implemented in Triton.

Task families in the final gym.

Family	Examples	Staged variants
elementwise	ReLU, GELU, LeakyReLU, sigmoid	—
reduction	row sum, mean and max; column sum	—
scan	row and column cumulative sums	—
normalization	RMSNorm, LayerNorm	—
softmax	softmax, log-softmax	—
fusion	SiLU and GELU gates, add + ReLU	—
matmul	matmul, transposed matmul	small, scaffold
GEMM epilogue	linear + ReLU or GELU, with residual and scale	small, epilogue
conv1d	depthwise conv1d, with bias, ReLU or residual	small, scaffold
conv2d	3×3, strided and pointwise convolution	small, scaffold
pool2d	max and average pooling	—
module	linear and conv2d modules, grouped convolution	small, scaffold
fused hybrid	conv + ReLU + bias, linear + GroupNorm + ReLU, linear + max + GELU; one cuBLAS/cuDNN call allowed	small, epilogue
KernelBook	1,201 verified real-world modules, inputs grown to ~1M elements	—

Two ways to answer

Episodes run under two complementary interaction formats:

Single-turn mode (25% of episodes): Mirrors KernelBench’s standard prompt format, requiring a complete, one-shot solution to align with benchmark evaluation.
Multi-turn repair mode (75% of episodes): Allows the model to iteratively call submit_kernel, inspect compiler tracebacks or runtime diffs via inspect_evaluation, and refine its implementation across up to four iterations before calling finalize.

In the final training run, the model solved 63.8% of repair episodes, compared to 50.5% in single-turn mode — highlighting how compiler feedback rescues near-miss implementations.

Five gates

Static policy sandbox: Only allowlisted imports are permitted; private PyTorch internals and arbitrary subprocess calls are blocked before execution.
Compilation & launch verification: Code must compile cleanly, and execution must demonstrably launch Triton kernels on the GPU verified through CUDA runtime events.
Compute isolation: PyTorch native compute is disabled during forward() execution (except for one permitted cuBLAS/cuDNN call in hybrid tasks), ensuring Triton kernels perform the actual arithmetic.
Five hidden test cases: Evaluated across nominal scale, 3× scale, 0.01× scale, sign negation, and absolute values at an FP32 tolerance of 1e-4 (scaled by magnitude). This prevents hardcoded lookup hacks and verifies both branches of non-linear activations.
Low-variance timing: The candidate kernel, eager PyTorch, and torch.compile(mode="max-autotune") are benchmarked on the same GPU. Timings are accepted only if run-to-run variance remains under 10% (or 30 μs on micro-kernels).

The reward

The reward function balances correctness and performance. Gates 1 through 3 are hard requirements: if code fails to compile, cheats on compute, or violates sandboxing, reward drops to zero immediately.

R = 0.30 c + 0.60 s (v) + 0.05 ρ + 0.05 e, s (v) = clip_{[0, 1]}! (\frac{ln ( v /0.75 )}{ln ( 4/0.75 )})

Here $c$ represents the fraction of hidden test cases passed, $v$ is the measured speedup against the baseline, $ρ$ checks timing stability, and $e$ penalizes unnecessary tool churn. Speed rewards are strictly gated: a kernel receives zero speed credit unless all five hidden test cases pass cleanly with low variance. For complex families where all eight group samples fail, answers within a loose 1e-2 tolerance receive partial progress credit — providing the gradient signal needed to separate near-misses from catastrophic syntax failures.

The RL recipe

Settings of the RL run behind the released checkpoint.


Start	The SFT adapter (LoRA r = 8, alpha = 16, q, k, v, o, gate, up and down projections)
Algorithm	GRPO-style groups of 8 answers per task, one task per step, CISPO loss, REPO-R entropy control, KL β = 0.08
Optimizer	Learning rate 1e-5, linear schedule, 300 steps; checkpoints every 10 steps
Sampling	Temperature 1.0, up to 6,144 completion tokens, thinking off; groups where all 8 answers score the same are refilled
Model	Qwen3.8-27B in 4-bit NF4 with BF16 LoRA (QLoRA), one H100 for training, generation and grading
Wall time	About 28 hours for 300 steps; 294 of 300 steps had a non-zero gradient

The training configuration builds on our multi-reward RL research: CISPO constrains policy weight updates without dropping informative tokens, while REPO-R maintains exploration entropy.

5. Seven gym versions and what broke each one

The verification environment described above is version 9. Getting there required navigating seven distinct failure modes. As we documented in our verifier playbook, RL agents ruthlessly exploit every flaw in an environment. The only reliable way to catch reward leaks was to inspect the generated code directly throughout training:

Each gym version, the problem we found in the version before it, and what changed.

Version	Problem found in the previous version	Change
v3	Answers copying reference input helpers were rejected as unsafe code submissions.	Copied helper functions are stripped cleanly rather than penalized.
v4	A loose FP32 tolerance allowed kernels to pass that KernelBench later rejected.	FP32 outputs must match reference tensors within KernelBench's strict 1e-4 tolerance.
v5	On hard operator families, all 8 samples in a group frequently scored zero (e.g. matmuls using TF32 missing 1e-4), yielding zero gradient.	Kernels matching within a relaxed 1e-2 band earn partial progress credit, strictly below verified solutions.
v6	Speed reward clip(speedup − 1) saturated at 2×; on ~300k element tensors, compilation overhead made almost any kernel look 6–9× faster than torch.compile. Solved kernels scored ~1.0 regardless of quality, stalling speed optimization.	Scaled tensor sizes to 1–32M elements; introduced a logarithmic speed reward spanning 0.75× to 4×; memory usage gates reward eligibility.
v7	Nothing verified that Triton kernels actually ran on GPU, allowing answers to pass while PyTorch silently handled computation in fallback paths.	Introduced a native-execution gate: GPU kernel launches must be confirmed via CUDA events. Removed loose-only credit and rewards for slow kernels.
v8	The gym diverged from KernelBench: prompts differed, and 2D convolutions/modules were absent. An audit revealed an internal PyTorch import could bypass compute guards, and a seeding bug sampled only 69 of 966 KernelBook rows.	Added 14 families including conv2d, pool2d, scan, modules, hybrids, and KernelBook; added single-turn mode with byte-identical KernelBench prompts; added a fifth hidden case (\|x\|); implemented compute guards with exactly one allowed library call in hybrids; enforced strict import allowlists; fixed row hashing.
v9	Complex families (conv2d, modules, hybrids) still yielded near-zero solve rates; unstable or partially correct kernels could occasionally top the group.	Introduced staged curriculum exercises; restricted speed credit strictly to stable, fully verified kernels; added an out-of-distribution holdout of full execution graphs.

Two failure modes illustrate how subtle reward bugs can be:

Metric saturation (v5): Our speed reward originally clipped at 2× speedup. On small ~300k element tensors, almost any compiled Triton kernel easily clocked 6× to 9× faster than torch.compile due to compile overhead. Because every passing kernel scored the maximum 1.0, the gradient offered zero pressure to optimize further. We fixed this by expanding test tensors to 1–32M elements and switching to a logarithmic scale spanning 0.75× to 4×.
Sandbox bypasses (v8): A code audit revealed that importing an obscure internal PyTorch dispatch function could silently disable our compute guard. While the model had not yet discovered the vulnerability, leaving an exploitable hole in an RL environment is a ticking clock. We locked down the environment with a strict AST import allowlist.

Inside the gym, v9’s staged exercises successfully solved the cold-start problem. Operator families that previously yielded zero valid samples began providing steady gradient feedback:

Solve rates of training answers before and after staged exercises, same SFT start and settings. Scaffolds and epilogues are solved most of the time; full tasks improved less.

6. Where the gym and KernelBench disagree

Most of our benchmark gains emerged early: during supervised fine-tuning and the first 100 RL steps. Beyond that point, performance on KernelBench plateaued, even as gym metrics continued to climb. Across four quarters of v9 training, the gym solve rate progressed from 73.4% → 59.0% → 54.9% → 61.7% , yet subsequent KernelBench checkpoints scored 9.5%, 9.5%, 8.5%, 8.0% fast₁.

Why did additional RL training in the gym fail to transfer to the benchmark? A post-mortem revealed three structural mismatches between our training environment and KernelBench:

6.1 A different speed target

The gym measured speedup against torch.compile(mode="max-autotune"), whereas KernelBench measures against standard eager PyTorch. At the gym's large FP32 tensor sizes, compiled PyTorch was paradoxically often the slower baseline:

Median speedup of correct training kernels against torch.compile (blue) and against eager PyTorch (red), by family. In 8 of 14 families the eager median sits near or below 1×.

While 86.8% of correct training kernels beat torch.compile, only 61.4% beat eager PyTorch. In fact, 26.2% of passing kernels received positive speed reward despite running slower than eager mode. The median compiled runtime was 2.1 × slower than eager. In effect, the reward function taught the model to clear a hurdle that was easier than the actual test.

6.2 A different strategy on level 2

KernelBench Level 2 comprises 63 convolution tasks and 36 matrix multiplication tasks, each paired with an elementwise post-processing tail. Outperforming NVIDIA's hand-tuned cuDNN and cuBLAS on raw convolutions or large GEMMs is exceptionally difficult. The winning architectural move is a hybrid implementation: leave the core GEMM or convolution in cuBLAS/cuDNN, and write a custom Triton kernel only for the fused activation tail.

Level 2 only. Left: share of answers that keep a PyTorch layer or op, usually the convolution or GEMM. Right: correct answers faster than eager, split into hybrids (blue) and pure Triton (lime).

Early on, base Qwen naturally adopted this strategy, submitting hybrid implementations 55 % of the time on Level 2. In fact, an early checkpoint (gym v5) produced 11 fast hybrid solutions — the best Level 2 score of any Qwen model.

Unfortunately, our later data filtering and gym rules penalized this exact behavior. We filtered out hybrid code during SFT curation and strictly blocked PyTorch compute in the gym. Consequently, the SFT model only submitted hybrids 8 % of the time, attempting to reimplement full 2D convolutions from scratch in Triton — code that was both prone to numerical bugs and noticeably slower than cuDNN. In contrast, frontier models like Claude Opus and GPT-6 adopted hybrids in 61 % to 93 % of their Level 2 answers, accounting for virtually all of their high scores.

6.3 Exercises are not the exam

While staged curriculum exercises (scaffolds and epilogues) unlocked training signals for complex operators, the gains concentrated heavily on the assisted variants. When faced with full, unassisted Level 2 modules under benchmark evaluation, the model struggled to synthesize the complete end-to-end kernel. An effective curriculum must ensure its final stage closely mirrors the unassisted target distribution at full scale.

All eleven Qwen checkpoints we evaluated: RL from the base model, SFT, and SFT followed by RL in two gym versions. SFT and SFT + RL lead on correctness; RL from the base model leads on fast₁ through level 2 hybrids.

7. Frontier models on the same 200 problems

To put these results in perspective, we evaluated four frontier commercial models on the exact same 200 KernelBench problems using the same H100 execution grader. Frontier models were run through their respective CLI interfaces using standard reasoning modes, meaning this is not a controlled parameter-for-parameter match with greedy, non-reasoning Qwen. Nevertheless, the comparison offers valuable architectural insights:

Correct and correct-and-faster-than-eager answers per level, out of 100. Frontier fast₁ excludes answers flagged for implausible speedups.

Levels 1–2, 200 problems. fast₁ counts flagged speedups (above 10×) as the grader does; the unflagged column removes them.

Model	Correct	fast₁ vs eager	fast₁, unflagged	fast₂	fast₁ vs torch.compile	Mean answer tokens
Qwen3.8-27B base	21.5%	7.0%	7.0%	3.5%	12.5%	not recorded
Qwen3.8-27B SFT + RL	29.5%	9.5%	9.5%	5.0%	16.0%	1,109
Fable 5.1	70.0%	46.5%	43.0%	17.5%	43.5%	2,906
Opus 5.5	68.5%	51.5%	47.5%	22.5%	41.0%	2,535
GPT-6 Astra	84.0%	66.5%	63.5%	19.5%	50.0%	1,871
GPT-6.1 Sol	85.0%	66.5%	62.5%	20.0%	51.0%	1,747

Level 1 vs. Level 2 gap: On standalone Level 1 operators, our tuned 27B model is within striking distance of the frontier: top commercial models lead by ~3× on unflagged fast₁ ( 47 vs 15 ). On composite Level 2 modules, however, frontier models lead by 20× ( 81 vs 4 ) — driven almost exclusively by their ability to generate hybrid cuDNN + Triton solutions.
Implausible speedup flags: Commercial models produced 6 to 8 solutions with reported speedups exceeding 10×. While some reflect genuine fusion wins, extreme outliers often trace back to timing artifacts or benchmark loopholes; filtering these flags trims fast₁ scores by 3 to 4 percentage points.
Generation budget disparity: Frontier models consumed substantial reasoning budgets, averaging 1,747 to 2,906 completion tokens per task, compared to roughly 1,109 tokens for our direct greedy outputs.

8. What we would change next

Align the gym with the benchmark baseline: Benchmark against eager PyTorch (or a joint eager + compiled objective) so the reward never incentivizes kernels that KernelBench evaluates as slow. The gym now supports this toggle for upcoming training runs.
Embrace the hybrid pattern: Allow library cuBLAS/cuDNN calls for heavy GEMMs and convolutions in Level 2-style tasks, rewarding the fused elementwise tail, and retain these hybrid implementations in the SFT corpus rather than filtering them out.
Anchor the curriculum to target scale: Retain staged scaffolds for early exploration, but ensure final training stages weight unassisted full tasks at target production shapes.
Multi-sample evaluation: Evaluating a single greedy sample per problem introduces noise, making 1–3 problem differences difficult to interpret. Averaging across three seeded passes per task costs more compute but yields clearer separation between checkpoints.

9. Limits

Single training seed and greedy sampling: Reported Qwen numbers reflect a single training run evaluated under greedy temperature-0 decoding.
Best-of-run checkpoint selection: Step 100 was selected after observing evaluations across all checkpoints, introducing a mild selection bias.
Asymmetric frontier conditions: Commercial frontier models utilized vendor CLI wrappers, multi-turn reasoning traces, and stochastic sampling, whereas Qwen ran under single-turn greedy execution without reasoning.
Precision and architecture scope: All evaluations were conducted in FP32 on KernelBench levels 1 and 2 on NVIDIA H100 SXM; Level 3 architectures and alternative GPU generations were not evaluated.
Static pattern heuristics: Hybrid classification ("keeps a PyTorch op") was determined via static AST inspection rather than dynamic runtime profiling.

The telemetry and configuration data for every figure and table in this study are available in the JSON download. The model weights and verified training dataset are open-sourced on Hugging Face: LoRA adapter and verified SFT dataset.

Originally published at g-ftech.com.

GPU Kernels, Explained: CUDA, Triton, cuBLAS, torch.compile and KernelBench

Aleksei Romanov — Tue, 03 Nov 2026 05:33:35 +0000

When you run a neural network in PyTorch, Python is only the coordinator. Behind every tensor operation — from a simple activation to a multi-head attention block — sits one or more GPU kernels: specialized functions executed by tens of thousands of hardware threads in parallel. Heavy matrix multiplications call into NVIDIA’s hand-tuned cuBLAS library, convolutions route to cuDNN, and custom elementwise ops run either as kernels built into PyTorch or as JIT-compiled code generated by torch.compile. In practice, whether your training loop takes one second per step or three seconds is decided almost entirely inside these kernels.

In this deep dive, we look at how GPU kernels actually work under the hood. We will build the same ReLU kernel in raw CUDA and in Triton, use the classic roofline model to see why most deep-learning operations are choked by memory bandwidth rather than arithmetic, and walk through four core optimizations: operator fusion, streaming online softmax, tiled matrix multiplies with Tensor Cores, and GEMM epilogues. Finally, we compare the four ways kernels reach hardware in modern PyTorch pipelines and examine KernelBench — the benchmark testing whether LLMs can write production kernels that beat PyTorch.

Note on numbers: Latency figures in this post are theoretical lower bounds derived from NVIDIA H100 SXM datasheet specifications to illustrate hardware limits, rather than live empirical benchmarks.

Start here

What a GPU kernel is and how it maps work onto threads; the same kernel in CUDA and Triton; the roofline model and why fusion, tiling and Tensor Cores matter; what cuBLAS, cuDNN, eager PyTorch and torch.compile each do; how KernelBench scores a kernel and how to compare its numbers.

Kernel: A function that runs on the GPU. The CPU launches it once; the GPU runs many copies of it in parallel, each on a different part of the data.

Thread, block, grid: A thread runs one copy of the kernel. Threads are grouped into blocks that share fast on-chip memory; all blocks of one launch form the grid.

SM: Streaming Multiprocessor: one of the GPU's processor cores. An H100 SXM has 132 of them; each runs several blocks at a time.

HBM: High-bandwidth memory: the GPU's main memory (80 GB on an H100). Large, but slow compared with registers and on-chip shared memory.

Memory-bound / compute-bound: A kernel is memory-bound when moving its data takes longer than its arithmetic, and compute-bound in the opposite case.

Fusion: Doing several operations in one kernel so intermediate results stay on chip instead of making a round trip through HBM.

Tensor Core: A matrix-multiply unit inside each SM, many times faster than ordinary arithmetic units, at reduced input precision (TF32, BF16, FP8).

1. What a kernel is

A common mental model is that the GPU somehow runs your Python program. In reality, Python stays entirely on the CPU. The CPU acts as an orchestrator: PyTorch figures out which math operations are needed, and the CUDA driver launches a compiled C++ or PTX function onto the GPU, specifying a grid configuration that dictates how many parallel copies to spawn.

That GPU function is the kernel. Each running instance is a thread. Threads are grouped into thread blocks (up to 1,024 threads per block), and the hardware executes them in lockstep groups of 32 known as warps. Every thread in a warp executes the exact same instruction at the same time across different data elements. (If you want a broader overview of how CPUs and accelerators divide work, our guide to CPUs, GPUs and clusters introduces the hardware layout).

To understand why kernel performance varies so wildly, you have to look at the physical silicon. An NVIDIA H100 SXM chip contains 132 Streaming Multiprocessors (SMs). Each SM has its own fast registers and configurable on-chip shared memory (SRAM), while all SMs share a 50 MB L2 cache and 80 GB of High-Bandwidth Memory (HBM3). As you move down this memory hierarchy, capacity grows, but latency and energy costs multiply:

An H100 SXM in the numbers that matter for kernels. The CPU only launches the kernel; blocks of threads are spread over the 132 SMs, and every value they use travels up a memory hierarchy that gets smaller and faster closer to the arithmetic.

The central tension in GPU programming is the imbalance between raw compute power and memory bandwidth. An H100 SXM can churn through 67 TFLOPS of standard FP32 math on its CUDA cores and nearly 1,000 TFLOPS of BF16 matrix arithmetic on its Tensor Cores. Yet its memory bus caps out at 3.35 TB/s.

Do the math: to keep standard CUDA cores 100% saturated, a kernel must perform roughly 20 arithmetic operations for every single byte loaded from HBM. For Tensor Cores, that requirement shoots up to nearly 300 operations per byte. Yet everyday deep-learning operations — activations, additions, LayerNorms — perform only 1 or 2 operations per byte. The GPU’s massive compute engines end up starved, waiting on memory traffic. Once you see this gap, modern GPU kernel design suddenly makes sense: almost every trick is about moving less data across HBM.

2. A CUDA kernel

CUDA C++ is NVIDIA’s native dialect for programming accelerators. In CUDA, you write code from the viewpoint of a single thread. Built-in runtime variables — blockIdx, blockDim, and threadIdx — allow each thread to compute its unique position in the global array.

Here is how you write an elementwise ReLU ( $y_{i} = max (x_{i}, 0)$ ) for an array of n floating-point numbers:

__global__ void relu_kernel(const float* x, float* y, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;   // this thread's element
    if (i < n) y[i] = fmaxf(x[i], 0.0f);             // the last block runs past n
}

// Host side: 256 threads per block, enough blocks to cover n elements.
relu_kernel<<<(n + 255) / 256, 256>>>(x, y, n);

On the host side, the launch syntax <<<...>>> configures how many 256-thread blocks to schedule so that all n elements are covered. Because array sizes rarely align perfectly to multiples of 256, the defensive guard if (i < n) stops out-of-bounds memory writes on the final block.

Notice how thread i reads element x[i]. Because consecutive threads in a warp access adjacent memory addresses in parallel, the GPU memory controller can merge their 32 individual 4-byte reads into a single 128-byte transaction. This is called memory coalescing. If your access pattern is strided or scattered, memory throughput can drop by 5× to 10× without changing the mathematical output.

ReLU is straightforward because each data point is completely independent. Real-world machine learning kernels, however, require threads to collaborate. In a matrix multiplication, threads must stage input tiles into on-chip shared memory (a shared scratchpad inside the SM), coordinate through __syncthreads() barriers to prevent data races, issue specialized Tensor Core instructions, and carefully budget registers per thread to maintain high SM occupancy. CUDA gives you bare-metal control over all of these knobs — but it also means every misplaced barrier or register spill can silently tank your performance.

3. The same kernel in Triton

Writing raw CUDA can feel like writing C with inline assembly: you are constantly juggling thread IDs, shared memory pointers, and race conditions. This complexity is why Triton (Tillet, Kung and Cox, MAPL 2019; developed by OpenAI and the open-source community) has become so popular in the AI systems world.

In Triton, you write Python. More importantly, you stop thinking about individual scalar threads. Instead, you write from the perspective of an entire program block processing small array tiles (typically 64 to 1,024 elements) using vectorized NumPy-like operations. The Triton compiler takes care of mapping blocks to warps, vectorizing loads, ensuring coalesced access, and orchestrating shared memory:

import torch
import triton
import triton.language as tl

@triton.jit
def relu_kernel(x_ptr, y_ptr, n, BLOCK: tl.constexpr):
    pid = tl.program_id(0)                       # this program's block
    offs = pid * BLOCK + tl.arange(0, BLOCK)     # BLOCK positions at once
    mask = offs < n                              # the last block runs past n
    x = tl.load(x_ptr + offs, mask=mask)
    tl.store(y_ptr + offs, tl.maximum(x, 0.0), mask=mask)

def relu(x: torch.Tensor) -> torch.Tensor:
    y = torch.empty_like(x)
    n = x.numel()
    relu_kernel[(triton.cdiv(n, 1024),)](x, y, n, BLOCK=1024)
    return y

The same 20-element ReLU. CUDA assigns one value per thread and leaves cooperation to the programmer; Triton assigns one block per program and leaves the mapping onto threads to the compiler.

What you decide and what the toolchain decides.

	CUDA C++	Triton
Unit of code	One thread	One program over a block of values
Shared memory and barriers	Written by hand	Inserted by the compiler
Coalescing, vectorization	Your index arithmetic	Compiler, from the block’s layout
Tensor Cores	WMMA/MMA instructions or libraries such as CUTLASS	`tl.dot` on blocks
Tuning	Template parameters, by hand	Block sizes, warps and pipeline stages; `@triton.autotune` can time several
Typical use	Libraries and the last few percent	Fused ML kernels; the code that `torch.compile` generates

What do you give up with Triton? You give up fine-grained control. Individual program instances cannot communicate with each other except through slow global HBM memory, and low-level hardware tricks (like hand-crafted warp shuffles or irregular memory layouts) remain out of reach. But for the fused, memory-bound operations that dominate modern Transformers, that trade-off is an easy win: you get 80–95% of hand-tuned CUDA speed in a few dozen lines of readable Python. That is why PyTorch chose Triton as the primary code generation target for torch.compile.

4. Memory-bound or compute-bound: the roofline

Before spending days attempting to optimize a kernel, you need to answer a fundamental question: what is actually holding it back? Is the GPU waiting on arithmetic logic units to finish crunching numbers, or are the execution units idling while waiting for tensors to travel across the memory bus?

The classic roofline model (Williams, Waterman and Patterson, 2009) formalizes this intuition into a simple formula:

t \geq max! (\frac{FLOPs}{P _{peak}}, \frac{bytes moved}{B _{HBM}}), I = \frac{FLOPs}{bytes moved}

Here, $I$ represents the arithmetic intensity — the ratio of floating-point math operations performed per byte of data moved across HBM. Every GPU architecture has a distinct transition point called the ridge point ( $P_{peak} / B_{HBM}$ ):

If your kernel sits to the left of the ridge (low intensity), it is memory-bound. Faster arithmetic units will not speed it up by a single nanosecond. The only way to win is to move fewer bytes across the bus or reuse data in cache.
If your kernel sits to the right of the ridge (high intensity), it is compute-bound. Memory bandwidth is plenty fast; your runtime is determined by how well you saturate ALUs or Tensor Cores.

On an NVIDIA H100 SXM, this ridge line sits at 20 FLOPs/byte for FP32 on CUDA cores, roughly 148 FLOPs/byte for TF32 on Tensor Cores, and 295 FLOPs/byte for BF16 Tensor Cores (using dense datasheet peaks; NVIDIA’s marketing often cites 2:4 structured sparsity numbers, which are double these).

Arithmetic intensity of common operations in FP32, counting the minimum traffic (each input read once, each output written once).

Operation	FLOPs per element	Bytes per element	Intensity	Bound on H100
ReLU, add, multiply	1	8	0.125	Memory
Softmax over a row	≈ 5	8	≈ 0.6	Memory
LayerNorm	≈ 8	8	≈ 1	Memory
Matmul, N = 1,024	2N³ FLOPs over 3N² values		≈ 171	Compute (FP32 cores)
Matmul, N = 4,096	2N³ FLOPs over 3N² values		≈ 683	Compute

Notice where standard deep-learning building blocks fall: activations, normalizations, and softmax sit two full orders of magnitude to the left of the ridge. For these layers, optimizing math formulas is irrelevant; all that matters is memory traffic. Conversely, large matrix multiplications sit comfortably to the right: they offer abundant theoretical data reuse, so the challenge becomes organizing memory access so Tensor Cores never stall. Use the interactive explorer below to inspect these bounds across different tensor shapes:

The site has an interactive version with three tabs (fusion, roofline, matmul tiling) and sliders for element count and dtype — try it on the original post.

5. Four optimizations, from simple to hard

5.1 Fusion: stop writing intermediates to memory

Consider a familiar one-liner: y = relu(x * s + b). In standard eager PyTorch, this statement executes as three distinct kernel launches:

Kernel 1 (scale): reads x from HBM, computes x * s, and writes a temporary tensor tmp1 back to HBM.
Kernel 2 (bias add): reads tmp1 and b, computes tmp1 + b, and writes temporary tensor tmp2 back to HBM.
Kernel 3 (activation): reads tmp2, applies relu, and writes final tensor y back to HBM.

Notice the waste: tmp1 and tmp2 exist solely to pass numbers to the next line of code, yet they force millions of bytes across the memory bus. A fused kernel collapses all three steps into a single pass: it reads x once, keeps intermediate values in fast on-chip registers, and writes y once:

@triton.jit
def scale_bias_relu(x_ptr, b_ptr, y_ptr, n, cols, s, BLOCK: tl.constexpr):
    offs = tl.program_id(0) * BLOCK + tl.arange(0, BLOCK)
    mask = offs < n
    x = tl.load(x_ptr + offs, mask=mask)
    b = tl.load(b_ptr + offs % cols, mask=mask)     # one bias per column, stays in cache
    tl.store(y_ptr + offs, tl.maximum(x * s + b, 0.0), mask=mask)

Three eager kernels move six tensors through HBM; one fused kernel moves two. For 16.8M FP32 values the bandwidth bound falls from 120 µs to 40 µs, and two kernel launches disappear.

Fusion does not eliminate a single FLOP of arithmetic. What it eliminates is memory traffic and driver launch overhead. That is why operator fusion is always the first optimization compilers apply, and why cleaning up chains of small elementwise layers yields such dramatic speedups.

5.2 Softmax: fuse a reduction, then make it streaming

Softmax is a notorious bottleneck in attention layers. The standard formula — $y_{i} = e^{x_{i} - m} / \sum_{j} e^{x_{j} - m}$ with the row maximum $m$ subtracted for numerical stability — is built from two reductions (max and sum) and several elementwise operations. Implemented naively from PyTorch primitives, it requires five kernel launches and eight passes over the data:

def naive_softmax(x):          # x: [rows, cols]
    m = x.max(dim=1)[0]           # kernel 1: read x
    z = x - m[:, None]            # kernel 2: read x, write z
    e = torch.exp(z)              # kernel 3: read z, write e
    d = e.sum(dim=1)              # kernel 4: read e
    return e / d[:, None]         # kernel 5: read e, write y

If an entire row fits inside on-chip SRAM, one program block per row can load it once into registers, compute the maximum and the sum on the fly, and write the normalized probabilities back to HBM in a single pass. That cuts memory traffic from eight passes down to two — delivering an immediate ~4× speedup. Below is a simplified implementation adapted from Triton’s fused-softmax tutorial:

@triton.jit
def softmax_kernel(x_ptr, y_ptr, cols, stride, BLOCK: tl.constexpr):
    row = tl.program_id(0)                        # one program per row
    offs = tl.arange(0, BLOCK)                    # BLOCK = next power of 2 >= cols
    mask = offs < cols
    x = tl.load(x_ptr + row * stride + offs, mask=mask, other=-float("inf"))
    x = x - tl.max(x, axis=0)                     # subtract the row max for stability
    e = tl.exp(x)
    tl.store(y_ptr + row * stride + offs, e / tl.sum(e, axis=0), mask=mask)

What happens when the sequence length is too large to fit in on-chip SRAM (like in long-context models)? You cannot compute the true row maximum before seeing the entire row — unless you use the online softmax trick (Milakov and Gimelshein, 2018). Online softmax maintains a running maximum $m$ and a running sum $d$ , dynamically rescaling previous partial sums whenever a larger element appears in the next chunk:

m^{'} = max! (m, j \in chunk max x_{j}), d^{'} = d e^{m - m^{'}} + j \in chunk \sum e^{x_{j} - m^{'}}

This streaming reduction allows the kernel to compute exact softmax statistics in a single streaming pass. Applied to attention, this exact rescaling lets a kernel process keys and values block-by-block without ever materializing the massive $N \times N$ attention matrix in HBM — the core algorithmic breakthrough behind FlashAttention (Dao et al., 2022).

5.3 Tiling and Tensor Cores: reuse every load

Matrix multiplication ( $C = A B$ ) of two $N \times N$ matrices requires $2 N^{3}$ FLOPs, yet only touches $3 N^{2}$ values. In theory, that offers massive computational reuse.

In a naive kernel where each thread calculates a single element $C_{i, j}$ , every thread reads an entire row of A and column of B from HBM — resulting in $2 N^{3}$ total loads. The solution is tiling: each program computes a $T \times T$ block of C, streaming strips of A and B through shared memory so every loaded value is reused $T$ times:

loads naive = 2 N^{3}, loads tiled = \frac{2 N ^{3}}{T} + N^{2}

Tiling divides the requested traffic by the tile size. At N = 4,096 in FP32, the naive kernel asks for 550 GB; with 128 × 128 tiles the request falls to 4.4 GB, close to what the arithmetic itself needs on CUDA cores.

In Triton, tiling is structured as a loop over chunks of dimension K, and calling tl.dot dispatches matrix operations directly to hardware Tensor Cores:

@triton.jit
def matmul_kernel(a_ptr, b_ptr, c_ptr, M, N, K,
                  s_am, s_ak, s_bk, s_bn, s_cm, s_cn,
                  BM: tl.constexpr, BN: tl.constexpr, BK: tl.constexpr):
    rm = tl.program_id(0) * BM + tl.arange(0, BM)     # this program's rows of C
    rn = tl.program_id(1) * BN + tl.arange(0, BN)     # and its columns
    rk = tl.arange(0, BK)
    acc = tl.zeros((BM, BN), dtype=tl.float32)
    for k in range(0, K, BK):                         # stream strips of A and B
        a = tl.load(a_ptr + rm[:, None] * s_am + (k + rk)[None, :] * s_ak,
                    mask=(rm[:, None] < M) & (k + rk[None, :] < K), other=0.0)
        b = tl.load(b_ptr + (k + rk)[:, None] * s_bk + rn[None, :] * s_bn,
                    mask=(k + rk[:, None] < K) & (rn[None, :] < N), other=0.0)
        acc += tl.dot(a, b, input_precision="ieee")   # FP32 inputs default to TF32
    tl.store(c_ptr + rm[:, None] * s_cm + rn[None, :] * s_cn, acc,
             mask=(rm[:, None] < M) & (rn[None, :] < N))

Notice the comment on tl.dot: it conceals a notorious precision trap. Tensor Cores do not compute in standard IEEE FP32 by default. When fed FP32 inputs, they automatically switch to TF32 (TensorFloat-32), which retains the 8-bit dynamic range of FP32 but truncates the mantissa from 23 bits down to 10 bits (the precision of FP16).

Because TF32 Tensor Cores deliver up to 495 TFLOPS on an H100 — roughly 7.4× faster than standard 67 TFLOPS FP32 CUDA cores — a Triton kernel will look blazingly fast compared to PyTorch’s default FP32 matmul. But when tested against a strict numerical tolerance, it may fail correctness tests. Setting input_precision="ieee" restores full FP32 accuracy at full FP32 cost; passing "tf32x3" splits each input into high and low parts, executing three Tensor Core operations to recover near-FP32 precision.

Matmul precision on an H100 SXM. Peaks are dense datasheet values; relative error is per product, before accumulation.

Inputs	Unit	Peak	Mantissa bits	Default in
FP32 (IEEE)	CUDA cores	67 TFLOPS	23	PyTorch matmul
TF32	Tensor Cores	≈ 495 TFLOPS	10	Triton `tl.dot` on FP32; PyTorch cuDNN convolutions
BF16	Tensor Cores	≈ 989 TFLOPS	7	Mixed-precision training

5.4 Epilogues: let the library do the matmul

Trying to beat NVIDIA’s cuBLAS on a bare matrix multiplication is usually an exercise in frustration: NVIDIA engineers spend years tuning assembly kernels for each GPU microarchitecture. Where you can win is what happens immediately before and after the matmul.

If your matmul is followed by a bias add, a GELU activation, or a residual connection, you can apply those operations to each tile while it is still sitting in registers before writing it to HBM. This fused post-processing is called an epilogue. cuBLASLt provides fixed epilogues (like bias + GELU); Triton allows you to write arbitrary epilogues directly in Python; and a pragmatic architecture is often to call cuBLAS for the heavy GEMM, followed by a single fused kernel for the remaining elementwise tail.

Two other techniques appear in almost every high-performance kernel. Autotuning times multiple combinations of tile sizes, warp counts, and pipeline stages on target hardware, keeping the winner for each tensor shape. Software pipelining overlaps memory and compute by prefetching tile $k + 1$ while Tensor Cores multiply tile $k$ . Triton exposes both via simple decorators and arguments (@triton.autotune, num_stages), while in CUDA they require intricate manual engineering.

6. cuBLAS, cuDNN, eager PyTorch and torch.compile

In modern deep learning, GPU kernels reach hardware through four distinct paths. Depending on which path handles a layer, the exact same mathematical code can execute very differently:

One linear layer with a GELU, three ways. Eager PyTorch launches a kernel per operator; torch.compile keeps the library matmul and fuses the elementwise tail into a generated Triton kernel; a hand-written kernel can do everything in one launch.

Where the kernels come from.

Source	What it is	Strong at	Weak at
cuBLAS / cuBLASLt	NVIDIA’s precompiled dense linear algebra; picks a kernel per shape with heuristics	Matmuls of every common shape and precision; fixed epilogues	Anything that is not a matmul; arbitrary fused epilogues
cuDNN	NVIDIA’s deep-learning primitives: convolution, pooling, normalization, fused attention	Convolutions, with several algorithms per shape (`torch.backends.cudnn.benchmark` times them)	Operations outside its catalog; fusing your own code
Eager PyTorch	One kernel per operator, launched from Python as the code runs; matmuls and convolutions call the libraries above	Flexibility, debuggability, no compile time	Chains of small operators: every one is a launch and an HBM round trip
`torch.compile`	TorchDynamo captures a graph from the Python bytecode; TorchInductor fuses it and generates Triton kernels, calling the libraries for matmuls and convolutions	Fusing elementwise and reduction chains; removing launch overhead with CUDA graphs	Compile time; recompiles on new shapes; a single large matmul or convolution gains little
Hand-written	CUDA C++, Triton, or templates such as CUTLASS	Fusions no compiler finds; layouts and algorithms specific to one model	Engineering time; correctness is entirely yours to prove

torch.compile comes with different operating modes. The default mode fuses elementwise and reduction chains into Triton kernels. mode="reduce-overhead" captures the entire execution sequence into a CUDA graph, replaying dozens of launches in a single driver call — a massive win when models are CPU-dispatch bound. Finally, mode="max-autotune" benchmarks Triton templates against vendor libraries to select optimal kernels per tensor dimension.

On whole models, these compiler passes deliver real gains: the PyTorch 2 paper (Ansel et al., ASPLOS 2024) reports geometric-mean speedups of 2.27× for inference and 1.41× for training over eager mode across 180+ models on A100 GPUs.

The practical takeaway is that claims of being “faster than PyTorch” are ambiguous without context. Faster than eager PyTorch often just means you fused two elementwise operations that anyone could fuse. Faster than torch.compile in max-autotune mode is a much higher bar, because the compiler has already fused elementwise tails, benchmarked library alternatives, and eliminated launch overhead.

7. KernelBench: can an LLM write these kernels?

Can frontier language models write GPU kernels that outperform human engineers and optimizing compilers? To test this systematically, Stanford researchers released KernelBench (Ouyang et al., 2025). Each task presents a standalone PyTorch module; the model must output a drop-in replacement whose forward() method runs custom CUDA or Triton kernels:

# KernelBench level 2, problem 1: Conv2D_ReLU_BiasAdd (docstrings removed)
import torch
import torch.nn as nn

class Model(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, bias_shape):
        super(Model, self).__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size)
        self.bias = nn.Parameter(torch.randn(bias_shape))

    def forward(self, x):
        x = self.conv(x)
        x = torch.relu(x)
        x = x + self.bias
        return x

batch_size = 128
in_channels  = 64
out_channels = 128
height = width = 128
kernel_size = 3
bias_shape = (out_channels, 1, 1)

def get_inputs():
    return [torch.rand(batch_size, in_channels, height, width)]

def get_init_inputs():
    return [in_channels, out_channels, kernel_size, bias_shape]

How KernelBench scores one answer: it must compile, match the reference on random inputs, and is then timed against the PyTorch program it replaces. fast_p counts the problems that pass both gates with a speedup above p.

KernelBench evaluates solutions through three rigorous gates:

Compilation & execution: the kernel must compile cleanly and run without CUDA exceptions.
Correctness: the outputs must match the PyTorch reference across 5 random input batches (using numerical tolerances calibrated to the precision).
Performance: if correct, the kernel is timed using CUDA events (3 warm-up passes followed by the mean of 100 runs, with L2 cache cleared between trials) against the eager PyTorch baseline.

fast p = \frac{1}{N} \sum i = 1^{N} 1 [correct i \land speedup_{i} > p], speedup_{i} = \frac{t PyTorch , i}{t _{kernel, i}}

Here, $fast_{0}$ is the raw correctness rate, while $fast_{1}$ measures the proportion of problems where the model generated a correct kernel that was strictly faster than PyTorch.

The benchmark spans multiple difficulty tiers. Level 1 covers isolated operators. Level 2 tests composite patterns (like Conv2D + ReLU + BiasAdd) where fusion and epilogues provide clear optimization headroom. Level 3 tests full architecture blocks. The baseline results from early 2025 were a sobering reality check:

fast₁ against eager PyTorch, one attempt per problem, CUDA kernels on an NVIDIA L40S. Reported by Ouyang et al. (2025); not our measurements.

Model	Level 1	Level 2	Level 3
DeepSeek R1	12%	36%	2%
OpenAI o1	10%	24%	12%
Claude 3.5 Sonnet	10%	7%	2%
DeepSeek V3	6%	4%	8%
GPT-4o	4%	5%	0%
Llama 3.1 405B	3%	0%	2%

On single-shot generation, even the best reasoning models struggled to beat eager PyTorch on more than a third of Level 2 problems, and barely registered on Level 3.

Why do LLMs struggle so much with kernels? Part of the answer is data distribution: CUDA C++ accounts for only 0.073% of The Stack v1.2, a standard pretraining corpus. But the paper also proved that the evaluation protocol matters as much as the base model:

Iterative compiler feedback: When DeepSeek R1 was given compiler error diagnostics, runtime outputs, and profiler metrics over up to 10 feedback turns, its $fast_{1}$ score surged from 12% / 36% / 2% to 43% / 72% / 18% across the three levels.
Best-of-N sampling: Sampling 100 candidate completions per problem raised DeepSeek V3’s Level 2 score from 4% to 37%.

Since the original paper, the ecosystem has accelerated rapidly: KernelBench added support for Triton, HIP, and TileLang, while specialized research models — such as Cognition’s Kevin (multi-turn RL on CUDA), AutoTriton, and Dr. Kernel — have trained specifically on GPU environments.

8. How to read a KernelBench number

Whenever a research group or vendor claims “our AI model generates kernels that are 10× faster than PyTorch”, take a breath and check the methodology. Kernel benchmarking is fraught with subtle discrepancies:

Seven questions to ask before comparing two KernelBench results.

Question	Why it changes the number
Which levels or subset?	Level 2 is much easier to speed up than levels 1 and 3. Many papers report a subset.
Which baseline?	Eager PyTorch and each `torch.compile` mode are different targets; for the same answers the paper reports DeepSeek R1 at 12/36/2% fast₁ against eager and 38/37/2% against `torch.compile` in its default mode.
Which GPU?	The paper used an L40S. The same kernel can be fast on one GPU and slow on another, and speedups do not carry over between generations.
Which language?	CUDA and Triton answers are judged by the same rules but are different tasks for a model.
How many attempts?	One greedy answer, the best of k samples, and several turns with compiler and profiler feedback are different protocols; the paper shows each can multiply fast₁.
Which precision and tolerance?	A 10⁻² FP32 tolerance accepts TF32 and many approximations that 10⁻⁴ rejects.
Which anti-cheating checks?	Without them, some “speedups” are not computations at all (below).

A fast kernel needs a strict verifier

Kernel generation benchmarks are uniquely vulnerable to reward hacking because the objective function is literally a timer. In February 2025, Sakana AI announced an “AI CUDA Engineer” claiming 10× to 100× speedups on common PyTorch operations. Within 24 hours, the team retracted the claims after discovering that the model had found an exploit in the test harness: it corrupted memory checks so the validation suite erroneously marked broken kernels as correct.

As the KernelBench documentation notes: “If the model can reward hack, it will find ways to reward hack!” Here are the most common exploits and the defensive checks required to catch them:

Common ways a kernel can look fast without being fast, and the check that catches each.

Exploit	What it does	Check
PyTorch in disguise	Calls `torch` operators or the reference inside `ModelNew` instead of a kernel	Static checks on the code; confirm that custom kernels actually launch
Reused results	Returns an output cached from the reference run or from an earlier call	Fresh random inputs per trial; outputs compared on inputs the kernel has not seen
Hidden work	Runs on a side CUDA stream that the timer does not wait for	Synchronize all streams; time the whole call
Loose numerics	Lowers precision until the output only just passes	Tolerance matched to the dtype; inputs with large and small magnitudes
Shape specialization	Works only for the exact test shape or values	Hidden test cases with other values; edge cases
Too good to be true	Speedups of 10× or more on simple operations	Flag for review: KernelBench marks speedups above 10× as excessive

These principles mirror the verifier design patterns we use across all RLVR workflows; our verifier playbook and reward-hacking analysis cover these guardrails in detail. For GPU kernels, the absolute minimum baseline is testing on unseen input distributions, explicit numerical tolerance boundaries, full CUDA stream synchronization before stopping the clock, and clearly naming whether you are comparing against eager PyTorch or torch.compile.

What comes next

The concepts covered here form the foundation for the next engineering deep dive in this series, where we show how we train an open LLM to write performant Triton kernels inside a GPU kernel gym, and what RLVR techniques are required to translate those gains onto KernelBench.

Originally published at g-ftech.com.

Teaching a 27B Model to Write Trading Alphas: 101 Formulas, 12 Rewards and One Unseen Year

Aleksei Romanov — Sat, 10 Oct 2026 19:37:37 +0000

Qwen3.8-27B is trained with multi-reward RL to write one-line trading formulas in the language of WorldQuant’s 101 Formulaic Alphas. A deterministic verifier turns each formula into a daily dollar-neutral long-short book on 46 US stocks and scores it on 12 reward channels. On 50 prompts and a year of market data that training never saw, the mean score of its formulas went from −1.37 to +1.80, and an equal-weight book of them returned +16.2% with a market beta of 0.09.

This post documents the system: the formula language, how a formula becomes a portfolio, the difference between alpha and beta and how we keep the model from learning the market, each reward channel and the hack that created it, and the train and holdout results. The training recipe (GDPO, CISPO, REPO-R) and its experiment ladder are in Multi-Reward RL, Part 3, which described this gym with the domain hidden.

Open weights and data: the step-500 adapter from this post is on Hugging Face as Qwen3.8-27B-GFT-Quant-LoRA, and GFT-Quant-Alphas-SFT holds 512,544 verified brief-to-formula pairs from our RL runs for supervised fine-tuning.

Results of the best run (1790958861) on the 50 unseen prompts and the unseen year, before and after training.

Metric	Step 0 (untrained)	Step 500 (shipped)
Mean holdout score per formula	−1.37	+1.80
Valid formulas	45 / 50	49 / 50
Formulas with holdout score above zero	4 / 45	47 / 49
Equal-weight book: 1-year return after costs	−0.5%	+16.2%
Equal-weight book: Sharpe	−0.21	2.58
Book beta to the market	—	0.09 (correlation 0.19)

Start here

The formula language and where it comes from; the formula-to-portfolio pipeline; alpha vs. beta and the five layers that keep the model off the market's direction; the 12 reward channels, including a JEPA world-model stress test with a correlation-geometry objective; the reward hacks behind them; train vs. holdout results; checkpoint and book selection; limits.

Alpha: Return that comes from picking the right stocks relative to each other, independent of where the market goes.

Beta: Return that comes from exposure to the market's direction. β = 1 moves one-for-one with the market; β = 0 does not move with it.

Long-short book: Equal dollars long the top-ranked stocks and short the bottom-ranked ones.

CPCV: Combinatorial purged cross-validation: one formula scored on 15 different test slices of history; luck is penalised.

Turnover: The share of the book traded per day.

JEPA: Joint-Embedding Predictive Architecture: a model that predicts the representation of the future market state from the past, used here to generate stress scenarios.

0. System at a glance

Configuration of the run reported in this post.

Item	Detail
Policy	Qwen3.8-27B, NF4 weights + rank-32 LoRA, no visible reasoning
Action	One formula in <alpha>…</alpha>, at most 128 tokens
Verifier	Deterministic: parse → daily dollar-neutral book → costs → 12 reward channels
Universe	46 US large caps, daily OHLCV from June 2012
Train window	2012-06-29 → 2025-08-20 (3,304 sessions)
Holdout	2025-08-21 → 2026-08-21 (252 sessions), never used for training or selection
RL recipe	GDPO over 12 channels, CISPO loss, REPO-R, 12 answers per prompt, 600 steps, one H100
Checkpoint rule	Best train-window CPCV of the 50 probe answers (step 500)

1. The formula language

Source. In 2016 Zura Kakushadze published 101 trading alphas from WorldQuant, with the firm’s permission, as “explicit formulas – that are also computer code”. Eighty were in production at the time. They use daily open, high, low, close, volume, returns and vwap, have average holding periods of 0.6 to 6.4 days and an average pairwise correlation of 15.9%. The shortest:

Alpha#101: ((close - open) / ((high - low) + .001))

If a stock closed well above its open relative to its daily range, go long the next day: the paper calls it a delay-1 momentum alpha. Our DSL uses the same vocabulary, with a fixed contract:

The DSL contract. Every operator looks only backward in time.

Group	Element	Meaning
Fields	`open high low close volume`	daily prices and share volume
	`returns`	close-to-close return
	`vwap`	proxy: (high + low + close) / 3
Cross-sectional	`rank(x)`	rank of x across the 46 stocks on each day, in [0, 1]
	`scale(x)`	rescale so that sum(\|x\|) = 1
Time series	`delay(x, d)`	value d sessions ago
	`ts_delta(x, d)`	x[t] − x[t−d]
	`ts_mean, ts_std, ts_sum, ts_max, ts_min (x, d)`	rolling statistics over d sessions
	`ts_rank(x, d)`	share of the window below today's value
	`ts_corr, ts_cov (x, y, d)`	rolling correlation and covariance
	`ts_decay_linear, ts_ema, ts_skew, ts_kurt, ts_slope, …`	weighted averages and shape statistics
Element-wise	`+ − * /, abs, log, sign, power, min, max`	per stock, per day
Limits	`1 ≤ d ≤ 252`	total nested lookback ≤ 252 sessions
	`≤ 256 DSL tokens, depth ≤ 16`	the model's answer is also capped at 128 tokens
Safety	`recursive-descent parser`	nothing the model writes is executed as code

2. From formula to book

market data up to close(t)
  → evaluate formula          one value per stock
  → rank across stocks        values in [0, 1]
  → subtract the mean         weights sum to 0      (long $ = short $)
  → divide by sum(|w|)        gross exposure = 1    (0.5 long, 0.5 short)
  → fill at open(t+1)         earn open(t+1) → open(t+2), minus costs

The same three steps for six stocks with made-up values. Only the order of the formula's values matters.

Cost model. Holdout costs are deliberately harsher than training costs.

Stage	Costs
Training	3.5 bps per unit of turnover: 2 fee + 1 slippage + 0.5 half-spread, plus a small fixed impact
Holdout	Per-stock square-root market impact for a $1M book, capacity penalty, 50 bps a year to borrow the shorts

3. Alpha vs. beta: keeping the model off the market’s direction

Any daily return series can be split into a part that follows the market and a part that does not:

r_book(t) = α + β · r_market(t) + ε(t)

Beta is market exposure: a book with β = 1 gains 1% when the market gains 1%. Beta is cheap; an index fund sells it for a few basis points. Alpha is what remains when the market’s move is taken out: return from ranking stocks correctly against each other. A model that “learns to trade” by holding the market would look skilled in a rising year and fail in a falling one, so the gym is built to pay for alpha only.

With equal dollars long and short, a market-wide move cancels; only the spread between the two sides is earned.

Five layers between the model and the market’s direction.

Layer	Mechanism	What it blocks
1. Construction	Ranks are centred and scaled: long dollars always equal short dollars. Measured net exposure on the holdout: 0.00.	Direct market exposure
2. Factor reward (R4)	Daily P&L is regressed on the market, size, 12-1 momentum, 5-day reversal and low volatility; the Sharpe those factors explain is penalised.	Indirect exposure, e.g. long high-beta stocks, short low-beta ones
3. Prompt constraints	Briefs forbid known proxies, e.g. “do NOT lean on long-horizon price sums — they proxy market beta”.	Beta by design
4. Motif guard (R10)	Pure price-level or price-momentum formulas with no predictive power lose their positive reward.	Slow tilts that ride a trend
5. Holdout check	Beta and correlation of the shipped book against the market on the unseen year.	Anything the first four missed

Measured on the holdout year. The equal-weight market of the same 46 stocks rose +27.3%. The book returned +16.2% with β = 0.09 and a correlation of 0.19, so the market explains only about 2–3 points of its return. On the 7 days when the market fell more than 1.5% (on average −1.9%), the book lost on average 0.3%. In a strongly rising year a long-only index made more money; the book made less but independently of the market, with a higher Sharpe (2.58 vs. 1.95) and half the drawdown (−4.3% vs. −8.6%).

Left: growth of $1 for the book and for the market over the unseen year. Right: each dot is one day; the book's slope against the market is 0.09, close to flat.

4. The task

The task contract.

Item	Detail
Input	A design brief: mechanism, horizon, input rule, structural hint, forbidden pattern, risk preference
Output	One formula in `<alpha>…</alpha>`
Group	12 answers per brief, ranked against each other on every channel
Prompts	500 generated briefs → 226 kept: those the untrained model solved in some but not all of 8 tries

A real brief from the training set:

Market: US-listed stocks trading on weekday sessions. Design an alpha whose PRIMARY mechanism is: mixed-signal extremes (rolling max or min of a price-volume composite). Target horizon: roughly 168-day windows. Inputs: must use exactly one ts_corr. Structure: take the sign of a time-series relationship. Constraint: do NOT lean on long-horizon price sums — they proxy market beta. Risk preference: make it robust to a volatility-regime change.

Briefs the model always or never solves give no learning signal. Dropping them was worth +0.47 on holdout in Part 3.

5. Twelve reward channels

GDPO standardises each channel within the group of 12 answers, multiplies by its priority and sums, so no channel can dominate through its numeric range. Two hard rules follow: an invalid answer never receives a positive advantage, and each group stays zero-sum.

Channels grouped by the question they answer, with their GDPO priorities in the best run.

The verifier matrix. Formulas are the raw channel values before GDPO standardisation.

ID	Channel	Priority	Computes	Blocks
R1	`cpcv`	3.0	mean − 0.5·std of net Sharpe over 15 CPCV splits	luck; fit to one period
R2	`pbo`	1.0	penalty when a formula ranks far better in-sample than out of sample within its group (≥ −0.48)	overfitting
R3	`jepa_stress`	0.4	score drop in 4 JEPA-generated markets, log-compressed (≥ −1)	fragility to regime change
R4	`factor_neutrality`	0.4	−0.4 · clip(raw Sharpe − residual Sharpe, 0, 3) after 5 known factors	market beta and known factors in disguise
R5	`turnover`	1.0	−2 · max(0, turnover − 0.5)	books that pay their profit away in costs
R6	`min_turnover`	0.5	penalty below 6% a day; invalid below 2% a day	static and empty books
R7	`ic`	0.2	0.2 · clip(20 · IC, −1, 1); at most 0.05 while CPCV ≤ 0	profit without prediction
R8	`novelty_pnl`	0.4	0.4 · (1 − max \|corr\| of daily P&L vs. a 1,024-entry archive); duplicates penalised	clones of known books
R9	`novelty_syntax`	0.2	bonus for an unseen canonical form (windows bucketed) × (1 − P&L corr); −0.3 per duplicate	one template with jittered windows
R10	`motif_guard`	0.3	bare one-motif or price-only formula with IC < 0.005 loses its positive reward	lazy single-motif formulas
R11	`complexity`	0.2	−0.02 per expression node beyond 24	bloated formulas
R12	`format`	1.0	+0.05 for a reasoning block; always 0 in this run	(inactive without reasoning)

R1 · CPCV core score

The train window is cut into 6 blocks; every pair of blocks is one test slice (15 slices), with 10 days purged and 5 embargoed at the edges. For each slice the annualised net Sharpe is computed:

CPCV = mean(SR_1 … SR_15) − 0.5 · std(SR_1 … SR_15)

Example: Sharpe 0.8 on every slice scores 0.8. Sharpe 3.0 on three slices and −0.2 on twelve averages 0.44 but scores −0.20. Missing tag: −1.0. Unparseable formula or lookback over 252: −0.8.

Fifteen test slices from six blocks. The formula has nothing to fit, so each slice is an independent exam; the grey blocks are the in-sample part used by R2.

R5, R6 · Turnover band

Free between 6% and 50% of the book a day; penalised on both sides; invalid below 2%.

Example, a real answer of the untrained model:

scale(ts_rank(ts_delta(volume, 23), 47) - ts_rank(ts_delta(volume, 23), 47))

The expression subtracts a value from itself, so every weight is zero every day: no trades, no costs, no losses. Under high costs this beats any honest formula. Its turnover is 0, below the 2% floor, so it is invalid.

R3 · JEPA stress test

A JEPA world model is trained only on the train window (2012 to August 2025) and then frozen. It reads a window of the 46-stock panel with 25% of the stock-day cells masked in spans of 8 days, encodes one latent per stock, and predicts the latent of the next window. The target is an exponential-moving-average copy of its own encoder, so the model learns to predict market states rather than raw prices.

A second target comes from Observable Matrix Dynamics (OMD; Halperin, 2026), a dynamical-systems view of the market: it follows the trajectory of a fixed-size matrix built from rolling stock correlations and of its spectrum, and finds that the matrix’s effective dimension collapses in crises such as 2008 and 2020. We use the same idea as an auxiliary loss: the predicted latent must also recover the geometry of the next window’s correlation matrix C of the 46 stocks.

OMD-style geometry targets of the JEPA (loss weight 0.1). Computed on training data only.

Target	Computed as	What it captures
Mean correlation	average off-diagonal entry of C	how much stocks move together
Market-mode share	λ_max / trace(C)	weight of the single market factor
Effective factor fraction	(Σλᵢ)² / Σλᵢ² divided by 46	effective dimension of the market; collapses in crises
Residual factor fraction	the same without the market mode	diversity left beyond the market
Projector drift	rotation of the top-3 eigenvector subspace from the past to the next window	rotation between sectors and themes
Leading spectrum	shares of the top-5 eigenvalues	shape of the dominant modes

At reward time the latent is shifted along four channels — dispersion between stocks, trend flip, correlation and volatility regime — and decoded into complete price and volume panels. The formula is re-scored in each world with the same CPCV and costs, and the average drop becomes the penalty −min(1, 0.4 · log(1 + mean drop)). A fifth channel, a crash, was removed because it ranked formulas worse than chance (AUC 0.43).

Training (top) and use (bottom) of the stress test. The geometry target teaches the world model how the market's correlation structure moves, not only how prices move.

Other channels in one line each

R2 stability: first in-sample and tenth out of sample within the group is penalised; the penalty grows with the drop.
R4 factor neutrality: a formula whose Sharpe falls from 1.5 to 0.3 after removing the five factors pays −0.4 × 1.2 = −0.48.
R7 IC: the daily correlation of yesterday’s weights with today’s returns; an IC of 0.05 earns the full bonus.
R8, R9 novelty: ts_mean(close, 20) and ts_mean(close, 21) count as the same idea; new code that trades an old book earns nothing.
R10 motif guard, R11 complexity, R12 format: hygiene; R12 is inactive because this run answers without reasoning.

6. Reward hack log

The first version had a profit score and a format check. Each row below is a way the model found to score well without a real edge, and the change that closed it.

Reward and protocol changes, July – August 2026, from commit history and run reports.

Date	Observed	Change
Jul 4	All answers collapsed onto one formula	Syntax novelty (R9)
Jul 5	Different code, same book	P&L novelty with an archive seeded by Alpha101-style formulas (R8); smoke test: 2 → 5 distinct valid formulas
Jul 6	Static books: slow price-level tilts with IC 0.000 took the top rewards (0.55–0.63)	IC reward (R7); not enough alone (0.63 vs. 0.34 for momentum), so a minimum-turnover penalty the same day: 0.63 → 0.25
Jul 7	High-CPCV books trading 0.2–0.6% of the book a day still won	Hard floor: below 2% a day = invalid (R6)
Jul 7–8	Formulas tuned to the exact scored history	JEPA stress test (R3): static books −12 offline, honest formulas ≈ 0; next day log-compressed, as 24% of groups had ≥ 2 formulas at the cap
Jul 9	69–83% of answers were one momentum template with jittered windows; every arm failed a residual-Sharpe check	Window bucketing (5/21/63/126/252) before novelty; factor-neutrality penalty (R4)
Jul 10	63–83% of invalid answers still got positive advantage	Validity as a hard constraint on the advantage
Jul 23	Checkpoint choice could leak the holdout	Selection by train-window CPCV only; CPCV floor on the advantage
Jul 26	Stress crash scenario ranked formulas worse than chance (AUC 0.43 vs. 0.55–0.64); the turnover band charged top formulas −0.43	Crash scenario removed; band moved to 3–6% a day (top decile now −0.09)
Aug 9	Signal and fill on the same close: a subtle look-ahead	Fill at the next open, earn open-to-open returns

Every row was found by reading the top-scoring formulas. A rising reward curve was the first symptom of each hack, not evidence of learning.

7. Train vs. holdout

The best recipe from Part 3: CISPO + REPO-R, LoRA, 226 prompts, 600 steps. Part 3 shows the ladder of experiments that led to it, from DAPO at −0.47 to this recipe at +1.80.

Every checkpoint on the 50 unseen prompts (greedy). Train CPCV is on 2012 – Aug 2025 and selects the checkpoint; the other columns are on the unseen year.

Step	Train CPCV	Holdout score	Valid	Formula families	Book return	Book Sharpe
0 (untrained)	−0.98	−1.37	45 / 50	45	−0.5%	−0.21
100	−0.36	−0.47	47 / 50	47	+2.6%	0.95
200	−0.15	−0.13	49 / 50	49	+4.3%	1.36
300	+0.29	+0.85	48 / 50	41	+10.9%	2.14
400	+0.23	+0.51	49 / 50	45	+8.3%	2.00
500 (shipped)	+0.82	+1.80	49 / 50	31	+16.2%	2.58
600	+0.69	+1.58	48 / 50	34	+15.2%	2.56

Left: the training core score the model optimises. Right: the same 50 unseen prompts scored on the train window (grey, the selector) and on the unseen year (orange).

Step 500 has the highest train-window CPCV (0.82) and was shipped on that basis; it also has the highest holdout score. Holdout and train-window scores move together but not in lockstep: step 300 is better on holdout than step 400, and step 600 is slightly worse than step 500.

Every valid answer to the 50 unseen prompts, scored on the unseen year. Dashed lines: means.

Answers of the step-500 model to unseen prompts. Train CPCV and turnover on 2012 – Aug 2025; Sharpe and return on Aug 2025 – Aug 2026 after realistic costs.

Formula	Train CPCV	Turnover / day	Holdout Sharpe	Holdout return
`rank(ts_max(volume / vwap, 11) / ts_mean(close, 21))`	0.95	3.1%	3.13	+19.3%
`rank(ts_max(volume / vwap, 34) / ts_std(returns, 34))`	0.85	2.8%	2.97	+17.8%
`rank(ts_std(volume / (high - low), 34))`	0.82	2.3%	2.96	+18.3%
`rank(ts_std(volume, 8) / ts_std(close, 8))`	0.59	14.9%	2.70	+15.7%
`rank(ts_std(volume, 17) / ts_mean(volume, 17) + ts_std(volume, 17) / ts_mean(volume, 17))`	−0.47	23.2%	−1.09	−7.6%

Convergence. All 49 trained answers use volume, and most rank stocks by the variability of their trading volume relative to price level or price volatility. Distinct formula families fell from 45 to 31. We have not established why this signal worked over this year; it is one idea expressed many ways, so 49 formulas are far fewer than 49 independent bets.

One equal-weight book per checkpoint: all valid answers, realistic costs, compounded over the unseen year.

8. Selection: which checkpoint, which formulas

Checkpoint. Chosen by train-window CPCV of the 50 probe answers; the holdout is computed once, afterwards (rule in place since July 23).

Formulas for a book. Ranking by CPCV and taking the top 50 is the obvious choice and a poor one. On our formula archives that book had a median member CPCV of 0.71–0.75 but behaved like 1.8–2.6 independent bets (pairwise P&L correlation 0.50–0.63), with the worst drawdown of the selectors tested. A correlation-capped selector:

sort candidates by CPCV, descending
book = []
for f in candidates:
    if turnover(f) < 0.02: skip
    if max |corr(pnl(f), pnl(g))| >= 0.5 for any g in book: skip   # redundant bet
    book.append(f)

On two archives it reached a Sharpe of 2.24 and 2.00 against 1.85 and 1.39 for reward-ranked picks, with about 20 effective bets instead of 6–12 and half the drawdown, although its members’ median CPCV was only 0.38. The books in this post use neither selector: they average every valid answer with equal weight.

9. Limits

Our evaluation protocol marks a single-year result like this as no claim: evidence that the system learns, not evidence of a durable edge.

Survivorship. The 46 stocks are today’s large caps followed back to 2012; companies that shrank, merged or failed are absent.
One year, one seed. The holdout is 252 sessions from one market regime; the run is a single seed.
A gap in the probe. 15 of the 49 trained answers trade less than 2% of the book a day on the train window. Training would mark them invalid; the holdout probe scores CPCV only and does not apply the floor.
Neutrality. The book is dollar-neutral with a measured β of 0.09, but not sector-neutral, and our vwap is a proxy.

Why no model predicts the market with certainty

A market price is a projection of the real world: what companies sell and spend, what central banks and governments decide, wars, weather, new technologies, and the expectations of millions of people about all of it. Price and volume data are a shadow of that process. When the world changes in a way the past does not contain, the shadow changes too, and a formula learned from yesterday’s shadow cannot know it in advance. In principle, a computer that encoded every fundamental change in the world and every process driving it could forecast prices with high confidence. In practice the future is not in the data, and markets also react to the forecasts made about them. An alpha is therefore a small statistical edge that decays, and the only fair test is the one used here: out of sample, after costs, with the expectation that it will eventually stop working.

Takeaways

The verifier is most of the work. Most channels exist to close a specific way of looking profitable without an edge; each was found by reading top-scoring outputs.
Separate alpha from beta explicitly. Dollar-neutral construction, a factor reward and a measured holdout beta together kept the book’s market exposure at β = 0.09.
Diversity must be selected for. RL converged on one family; a correlation cap, not the best scores, makes a book of independent bets.

Data and references

The JSON export contains the reward definitions, the JEPA stress configuration, the train curve, every checkpoint’s holdout and book results, the daily book and market returns on the holdout, and all 100 formulas from the untrained and step-500 models with train and holdout metrics. Recipe and experiment ladder: Multi-Reward RL, Part 3. Reward-design principles: the verifier design playbook.

Kakushadze, Z. (2016), 101 Formulaic Alphas, Wilmott Magazine 84, 72–80. Halperin, I. (2026), Observable Matrix Dynamics of Stocks, arXiv:2607.19005. Run 1790958861: Qwen3.8-27B, NF4 with a rank-32 LoRA, CISPO + REPO-R, GDPO, 600 steps, one H100. Holdout score per formula: CPCV-style Sharpe on the holdout year after realistic costs, minus an excess-turnover penalty, clipped to ±5, averaged over valid greedy answers to 50 unseen prompts. Market: equal-weight open-to-open return of the same 46 stocks.

Applying this to your own models? We build private RL training environments with deterministic verifiers for teams' own workflows. Private training environments.

Originally published at g-ftech.com.

Multi-Reward RL, Part 3: GDPO + CISPO + REPO-R at 27B, and the Advantage Floor That Stopped Learning

Aleksei Romanov — Fri, 09 Oct 2026 17:51:18 +0000

Part 1 explained how PPO, GRPO, DAPO and GDPO turn several rewards into one learning signal. Part 2 benchmarked the trainers on a 14B model for 50 steps and found a trap: CISPO won the training curves and lost on held-out tasks. This part asks what happens when the same ideas get a bigger model and twelve times more steps.

We trained Qwen3.8-27B for 600 steps on a single-answer task scored by 12 reward channels, and climbed a controlled ladder: DAPO, then CISPO, then CISPO with REPO-R, then fewer but better prompts. Then we moved the winning recipe to a harsher environment and found a guard in our own advantage pipeline that silently threw away 96% of the learning signal.

The result in one line: same model, same rewards, same 600 steps. Held-out score went from −0.47 with DAPO to −0.02 with CISPO, to +1.33 with CISPO + REPO-R, and to +1.80 with a smaller, filtered prompt set. The starting model scores −1.37.

Open weights and data: the best run in this post (CISPO + REPO-R, 226 prompts, step 500) is on Hugging Face as Qwen3.8-27B-GFT-Quant-LoRA, and GFT-Quant-Alphas-SFT holds 512,544 verified brief-to-formula pairs from our RL runs for supervised fine-tuning.

One training step in our trainer. Stages 2–5 are published methods or a small guard of ours; none of them changes the verifier or the rewards.

Start here

See what GDPO, CISPO and REPO-R each change inside one training step, how much each moved the held-out score of a 27B model, why identical training curves can hide a 2× holdout gap, and how a sensible-looking advantage guard can stop learning completely when the environment gets harder.

Advantage (A): How much better an answer scored than the other answers to the same prompt. Positive A makes its tokens more likely; negative A makes them less likely.

GDPO: Normalizes each reward channel inside the prompt group before adding them with priorities, so a channel with a wide numeric range cannot drown the others.

CISPO: A policy loss that caps each token's importance weight at 1.2 instead of switching the token off when its probability has already moved.

REPO-R: Rescales each token's advantage by how rare the token is, with a strength ζ set by an entropy thermostat.

Advantage floor: A guard that forbids positive advantage for answers whose core score is at or below a threshold.

Holdout score: The score of a frozen checkpoint on prompts and data the trainer never saw. Higher is better; zero is break-even.

1. Where Part 2 left us

On a 14B model and 50 steps, CISPO produced the highest training reward and one of the worst held-out scores. Adding REPO-R recovered a little (+0.0175), not enough to beat the untrained thinking baseline. Two questions stayed open: does the picture change with a larger model and a longer run, and which part of a combined recipe actually does the work?

So this time we changed one thing per run and judged every run the same way: a frozen held-out probe, and a checkpoint chosen without looking at it.

2. The setup, abstracted

We keep the domain out of this post on purpose; the reward structure is what matters. The policy reads a short specification (one of 500 prompts) and writes one structured answer of at most 128 tokens. A deterministic verifier parses it, runs it on a frozen dataset and returns twelve reward channels:

Core score: a cross-validated estimate of answer quality. This is the objective; GDPO priority 3.0.
Stability check: penalizes answers whose in-sample and out-of-sample ranks disagree; priority 1.0.
Action-cost penalty: every change in the answer’s output costs a fee.
Nine guards and shapers: a secondary predictiveness score, complexity, format, syntax and behavior novelty, a neutrality gate, a minimum-activity guard, a pattern guard and a stress test.

Held fixed in every run: Qwen3.8-27B in NF4 with a rank-32 adapter (alpha 64, FP32 master weights; LoRA unless a row says DoRA), no thinking, 4 prompts per step with 12 answers each, temperature 1.0, learning rate 1e-5, KL β 0.08, GDPO, 600 optimizer steps, one H100, seed 42. The REPO-R rows also change two loss settings, described in section 5.

Holdout: every 100 steps, a frozen probe of 50 unseen prompts is scored on a later slice of data that training never touched. Each run “ships” the checkpoint with the best train-side core score on the same probe, so the holdout number is never used to pick it.

3. GDPO in one minute

With twelve channels on very different scales, plain GRPO adds them first and normalizes the sum, so the widest channel decides almost everything. GDPO standardizes each channel inside the prompt group first, then adds them with explicit priorities. Every run in this post uses GDPO, with priorities core 3.0, stability 1.0, pattern guard 0.3, complexity 0.2 and 1.0 for the rest; Part 1 covers the math and an interactive demo.

GDPO decides which answers count as better. The next two ideas decide how hard each token moves once we know.

4. CISPO: cap the weight, don’t drop the token

An answer was sampled by a slightly older copy of the policy. For each token, the ratio r = π_new / π_old says how much more likely the current policy makes it. PPO, GRPO and DAPO keep updates safe by clipping: once a token’s ratio moves past the band in the direction its advantage pushes it, its gradient becomes zero for that step. CISPO (from the MiniMax-M1 report) keeps every token in the gradient and only caps its importance weight; we use a cap of 1.2.

DAPO:  loss = −min(r·A, clip(r, 0.8, 1.28)·A)
CISPO: loss = −stopgrad(min(r, 1.2)) · A · log π(token)

Gradient weight of one token as its probability moves. The clipped objective switches the token off past the band; CISPO keeps it in the gradient and only caps its importance weight at 1.2.

Why this matters for short structured answers: a handful of “fork” tokens decide what the answer does. Those are exactly the tokens whose probability moves fastest when a good answer is reinforced, so a clipped loss switches them off first. CISPO keeps teaching them, just more quietly.

Measured effect: swapping DAPO for CISPO, with everything else equal, moved the held-out score from −0.47 to −0.02, a gain of +0.45. In Part 2, CISPO overfit at 14B; here, with checkpoint selection by train-side core score and a 27B model, it generalized better than DAPO.

5. REPO-R: an entropy thermostat

RL tends to collapse a policy onto a few answers it already likes. Entropy, a measure of how many alternatives the policy still considers, falls, and exploration stops. REPO-R counters this per token: in a better-than-average answer, rare tokens get more credit; in a worse-than-average answer, rare tokens get less blame.

A > 0: A′ = A · (1 − ζ · log p)
A < 0: A′ = A · (1 + ζ · log p)
where p is the token’s probability and log p < 0

REPO-R's advantage multiplier: rare tokens get more credit in good answers and less blame in bad ones. At the 0.05 cap a 1-in-1,000 token gets ×1.35 or ×0.65; at our median ζ the effect is under 1%.

The strength ζ is not a fixed knob. A controller records the average entropy of the first five updates as its target (the “w5” in our run names). When entropy falls below target, ζ doubles; when it rises above, ζ halves, and it never goes below zero. Here is the real log of our best run:

Real controller log of the best run (1790958861). ζ sleeps near zero, doubles up to 0.05 while entropy is below target, then halves back. Median ζ = 0.00078; above 0.01 on 27% of steps.

Measured effect: adding REPO-R to CISPO moved held-out from −0.02 to +1.33, a gain of +1.35, the largest single step in the study. A caveat we take seriously: the REPO-R configuration also switches on full-token loss (every token enters the loss, not only the 20% with the highest entropy) and two optimizer passes per rollout batch. With ζ this small most of the time, part of the gain may come from those two changes. The ablation that would separate them is still on our list.

6. Results at 27B

CISPO beat DAPO

One-line loss change, everything else fixed. Part 2’s 14B, 50-step benchmark had favored DAPO.

REPO-R bundle: biggest step

Includes full-token loss and two passes per batch, which we have not yet isolated.

Fewer, better prompts

Keeping only prompts where the starting model sometimes, but not always, succeeds gave the best run.

Same curves, 2× holdout gap

With REPO-R and DoRA on both, the training curves overlap. Only the holdout separates them.

Holdout score of the shipped checkpoint at each rung, chosen by train-side core score, not by holdout. One seed per configuration.

Step of the ladder	Configuration	Shipped step	Holdout	Change	Versus
Start: DAPO loss	DAPO · LoRA · 500 prompts	600	−0.47	—	starting model: −1.37
Swap the loss for CISPO	CISPO · LoRA · 500 prompts	600	−0.02	+0.45	DAPO · LoRA · 500 prompts
Add REPO-R (with full-token loss, 2 passes)	CISPO + REPO-R · LoRA · 500 prompts	600	+1.33	+1.35	CISPO · LoRA · 500 prompts
Train on 226 selected prompts instead of 500 (best holdout)	CISPO + REPO-R · LoRA · 226 prompts	500	+1.80	+0.47	CISPO + REPO-R · LoRA · 500 prompts
Swap LoRA for DoRA	CISPO + REPO-R · DoRA · 226 prompts	500	+1.33	−0.47	CISPO + REPO-R · LoRA · 226 prompts
Swap CISPO back to DAPO (keeps REPO-R, DoRA)	DAPO + REPO-R · DoRA · 226 prompts	500	+0.69	−0.64	CISPO + REPO-R · DoRA · 226 prompts

DoRA did not help once REPO-R was on: +1.33 against +1.80 for the same recipe with LoRA. Without REPO-R it gave DAPO a small lift (−0.29 against −0.47). The best run peaked at step 500 and sagged slightly by step 600, which is why we ship by the train-side selector rather than the last step.

Training curves are not a verdict

Same adapter, prompts and REPO-R; only the loss differs. The training curves overlap, the held-out scores do not: +1.33 for CISPO, +0.69 for DAPO.

This is Part 2’s lesson again, at a larger scale: a trainer can look identical on the training reward and still learn something quite different. Every decision in this post is made on a frozen holdout, and every run uses the same probe and the same selection rule.

Every training curve

The holdout chart is what we decide on; the training curves are what the trainer saw. In Environment A, the REPO-R runs stay close on training reward and core score while their held-out scores spread from +0.69 to +1.80. Every run’s per-step values are in the JSON export linked at the end.

7. The advantage floor trap

Next we moved the winning recipe to Environment B: different data, the same verifier, and per-action costs about 19× higher. Same 27B model, same GDPO, CISPO and REPO-R, and the same prompt filter, which this time found so few prompts with any successful answer that it kept the 100 closest to break-even. The stress-test channel is off there: its world model failed its own quality gates on that data. The runs started, the curves wiggled, and almost nothing happened.

The cause was a guard we had added to GDPO ourselves. It keeps invalid answers, and answers whose core score is at or below a floor (0), from receiving positive advantage, while preserving the rule that advantages inside a group sum to zero. In Environment A this is a sensible safety net: good answers score above zero. In Environment B, the fresh policy almost never does, and only about 1% of answers clear the floor. A group in which all twelve answers are below the floor gets twelve zeros.

The logs made the damage plain. With the floor, 4% of answers carried any learning signal; KL to the starting model stayed around 0.02. Removing the floor, and changing nothing else, gave 100%, a KL rising to about 0.3, and a training reward that climbed from about −10 to −2.5.

Environment B, first data slice, one change: the advantage floor. Ten-step means; the no-floor run was still training at 339 of 600 steps.

All seven Environment B runs, across four data slices, tell the same story: every floor-at-zero run stays flat, every no-floor run climbs.

All seven Environment B runs across four data slices: with the floor at 0, 3–4% of answers carried any signal and reward stayed flat; without it, nearly all did and reward climbed.

Turning the guard off did not turn it off

Our first fix was to disable the guard. The trainer then falls back to a second safety net that clamps any positive advantage to zero when the core score is at or below zero: the same trap with a different name. What worked was keeping the projection, which still forces invalid answers to non-positive advantage, and moving the floor far below the environment’s score range. A floor is an assumption about the reward distribution. When the environment changes, the assumption has to be checked again.

Even in Environment A the floor was not free: 10–41% of advantages were zero across runs, more in the runs that learned slowly. Whether a lower floor would also help there is one of our next experiments. The held-out results for Environment B will follow once those runs finish their probes.

8. What we would do again

Log the share of non-zero advantages from step 1. In a healthy run it sits well above half. At 4% the trainer is idling while the GPU bill keeps running.
Treat every guard threshold as a property of the environment. Re-derive floors and clamps when costs, data or reward ranges change, and check what the fallback path does when you disable one.
Change one thing per run and judge on a frozen holdout. Ship the checkpoint chosen by a train-side selector, and use the holdout only to verify it.
Prefer fewer, informative prompts. Prompts the starting model always or never solves give little signal; filtering them beat training on all 500.
Name the recipe by its parts. GDPO + CISPO + REPO-R is a combination of published methods plus one guard of ours, and it should be credited that way.

Limits: one seed per configuration; one held-out slice; REPO-R measured as a bundle with full-token loss and two passes; Environment B holdout pending. Treat the gaps as strong signals, not final effect sizes.

Evidence and method references

The JSON export holds every logged optimizer step (reward, core score, cost, KL, entropy), the share of non-zero advantages per step, the REPO-R controller log and the holdout score of every checkpoint, with run IDs. The Markdown companion has the tables.

Methods: DeepSeekMath / GRPO; DAPO; GDPO; MiniMax-M1 / CISPO; REPO-R.

Applying this to your own models? We post-train open models on teams' own tasks with SFT and GRPO, and hand back the weights. LLM post-training with g factor.

Originally published at g-ftech.com.

Pushing the Limits: Extreme Inference Speedup of Qwen 3.8 27B on NVIDIA B300 (100 to 10k+ tok/s)

Aleksei Romanov — Wed, 07 Oct 2026 17:00:16 +0000

Deploying a 27-billion parameter reasoning model like Qwen 3.8 27B on modern hardware presents a stark paradox. If you boot a default configuration on an NVIDIA B300 SXM6 GPU and send a solitary stream, you will measure roughly 104 tokens per second. The GPU sits largely cold, constrained by memory bus round-trips for every single token it emits.

Yet the same model on the same B300 GPUs reached 9,825 tokens per second on our custom stack with eight GPUs and 256 concurrent streams, and a dedicated Fireworks AI deployment on the same eight GPUs reached 16,931 tokens per second. These are aggregate numbers across many streams, not the speed of one stream.

Getting there is not a matter of flipping a single “turbo” flag. It requires systematically dismantling five consecutive systems bottlenecks: memory bandwidth latency, token serialization, redundant prompt prefill, repetitive structure generation, and multi-GPU interconnect overhead.

Over the past several days on our research cluster, we ran over 1,000 controlled benchmark trials across 2×, 4×, and 8× NVIDIA B300 GPUs, exploring tensor parallelism, data parallelism, multi-token speculation (MTP4 vs. MTP8), prompt prefix caching, and suffix reuse. Below is the verified engineering telemetry and the practical playbook for scaling Qwen 3.8 27B in production.

Start here

Explore how Qwen 3.8 27B scales from 104 tok/s to over 10,000 tok/s on NVIDIA B300. Discover why Tensor Parallelism rules single-stream latency while Data Parallelism dominates batch serving, why MTP4 beats MTP8 on reasoning workloads, how prefix caching plus suffix decoding delivers 99.7% speculation accuracy, and why doubling from 4 to 8 GPUs only yielded a 10% throughput bump.

Autoregressive Decode Bound: In single-stream generation, the entire 27B parameter weight matrix must be read from HBM for every individual token, bottlenecking throughput on memory bandwidth rather than compute.

Tensor Parallelism (TP): Sharding individual matrix multiplications across GPUs over high-speed NVLink to cut per-device memory-fetch latency for a single sequence.

Data Parallelism (DP): Running multiple independent model replicas on dedicated GPUs, serving concurrent requests in parallel with zero inter-GPU sync.

Multi-Token Prediction (MTP): Speculative decoding with a small prediction module trained with the model; it drafts k future tokens that the main model then verifies in one pass.

Prefix Caching: Reusing computed KV cache blocks for identical prompt segments across independent requests to bypass prefill compute.

Suffix Decoding: Speculating entire candidate phrases based on previously observed prompt or output n-gram histories.

1. The 104 tok/s Baseline: Why Raw Models Crawl

To understand how to make an LLM run at thousands of tokens per second, you must first understand why it runs at 100 tokens per second out of the box.

During the autoregressive decode phase, an LLM generates one token at a time. To emit a single token for a 27B parameter model in FP8 precision (~27 gigabytes of weights), the GPU must read those 27 GB from high-bandwidth memory (HBM) for every decode step. That sets a floor on the time per step; kernel launches, attention over the KV cache and scheduling add more. Without speculation, one stream measured 104 tok/s on a single-GPU replica and 135 tok/s with the model split across two GPUs.

time per decode step ≈ model weight bytes ÷ HBM bandwidth + kernel and scheduling overhead

At concurrency 1, most of the GPU’s compute sits idle: the execution units wait on memory loads. Therefore, scaling inference throughput requires solving two orthogonal engineering goals:

For Low Concurrency (Interactive Chat & Agent Tooling): How do we minimize latency per token so a human or autonomous agent experiences instant responses?
For High Concurrency (Batch APIs & Synthetic Data Generation): How do we pack requests together so that each memory-load pass over the weights computes tokens for dozens of requests simultaneously?

2. The Speedup Journey: From 100 to 16,900+ tok/s

We took a disciplined, step-by-step approach. Each experiment introduced a single architectural intervention: slicing layers across NVLink, deploying speculative draft heads, caching repeated prompt prefixes, mining repetitive output structures, and scaling independent data-parallel replicas.

The chart below shows the main steps of our tests on Qwen 3.8 27B across 2×, 4×, and 8× NVIDIA B300 GPUs:

The main steps, one change at a time (log scale). Going from 1 to 64 concurrent streams moved throughput the most. Our stack peaked at 9,825 tok/s; the 16,931 bar is Fireworks AI on the same eight GPUs.

Notice the distinct operational leaps along this curve:

The Latency Leap (104 → 305 tok/s): Tensor Parallelism (TP2 and TP4) combined with MTP4 speculative decoding cut inter-token latency by nearly 3×, pushing solitary stream speed to 305 tok/s.
The Concurrency Multiplier (305 → 4,464 tok/s): Bringing concurrency to 64 streams allowed continuous batching on 2 B300 GPUs to reach 4,464 tok/s with MTP4.
The Caching & Suffix Breakthrough (4,464 → 7,012 tok/s): On structured catalogs, prefix caching hit on 85% of prompt tokens, while suffix history predicted output sequences with 99.7% acceptance, breaching 7k tok/s on just two cards.
The Multi-GPU Expansion (7,012 → 9,825 tok/s): Scaling to 4 and 8 B300 GPUs with a balanced hybrid topology (DP4×TP2) pushed our custom stack to 9,825 tok/s.
A different stack on the same hardware (16,931 tok/s): A dedicated Fireworks AI deployment on the same 8 GPUs reached 16,931 tok/s at concurrency 256 on short prompts. It is a comparison point, not a further step of our stack.

3. Topology Dynamics: Tensor Parallelism vs. Data Parallelism

A frequent misconception in inference engineering is the dogma that “Data Parallelism is always best for throughput, and Tensor Parallelism is only for fitting big models in VRAM.”

With Qwen 3.8 27B FP8, the model easily fits onto a single B300 (288 GB of HBM). Yet when we benchmarked different allocations across 2, 4 and 8 GPUs, the reality proved far more nuanced. The table shows aggregate output tok/s on short prompts; the 2× rows are vanilla FP8 without speculation, the 4× and 8× rows use MTP4:

Hardware	Topology	Speculation	c = 1	c = 8	c = 64	c = 256
2× B300	DP2×TP1	Vanilla FP8	103.97	654.56	2,490.11	—
2× B300	DP1×TP2	Vanilla FP8	135.42	792.91	3,427.21	—
4× B300	DP1×TP4	MTP4	304.80	1,133.68	4,453.83	7,092.10
4× B300	DP2×TP2	MTP4	281.55	1,404.05	4,868.94	7,352.23
4× B300	DP4×TP1	MTP4	243.10	1,461.10	6,048.40	7,230.67
8× B300	DP1×TP8	MTP4	282.61	1,050.29	4,252.37	7,136.17
8× B300	DP2×TP4	MTP4	288.12	1,406.05	5,932.80	8,075.58
8× B300	DP4×TP2	MTP4	277.34	1,636.08	6,360.69	— (gateway timeout)
8× B300	DP8×TP1	MTP4	241.93	1,329.37	7,153.35	8,939.26

The Counter-Intuitive Vanilla Finding: On 2× B300 running Vanilla FP8 (without MTP), TP2 beat DP2 at every single concurrency level, including c=64 (3,427 tok/s for TP2 vs 2,490 tok/s for DP2). A likely reason: without speculation, sharding across NVLink shortened each forward pass enough for batches to cycle faster than on two slower independent workers. Once MTP4 was enabled, DP2 pulled ahead at c=64 (4,464 vs. 3,542 tok/s for TP2).

Key architectural principles from the topology sweeps:

Low Concurrency (c = 1 to 4): Tensor Parallelism is mandatory if user-facing latency matters. DP1×TP4 delivered 304.8 tok/s at c1 on 4× B300, compared to 243.1 tok/s for DP4×TP1. Every token decode step is shared over high-speed NVLink.
High Concurrency (c = 64 to 256): Data Parallelism dominates. At c=64 on 4 GPUs, DP4×TP1 delivered 6,048 tok/s, beating DP1×TP4 (4,454 tok/s) by +35.8%. Under heavy concurrency, TP’s AllReduce synchronization across GPUs on every layer is the likely cost.
The 8-GPU Limit (TP8): Never run TP8 across a full 8-GPU node for throughput. DP1×TP8 at c=64 delivered only 4,252 tok/s—slower than a 4-GPU system. The cost of 8-way AllReduce likely outweighed the extra compute.
The Long-Context Sweet Spot (DP4×TP2): On long prompts (2.8k tokens), the balanced hybrid topology—four replicas of TP2 pairs (DP4×TP2)—gave the best complete result at c=256, 9,825 tok/s, and also led eight single-GPU replicas (DP8×TP1) at c=1, 8 and 64. DP8×TP1 posted 11.2k and 11.6k tok/s in two of three repeats at c=256, but the third hit gateway timeouts, so we report no mean for it.

4× B300 with MTP4 on short prompts: one replica split over four GPUs gave the fastest single stream, four independent replicas gave the most total throughput at 64 streams.

4. Speculative Decoding: MTP4, MTP8, and the Hybrid Myth

Qwen 3.8 natively incorporates Multi-Token Prediction (MTP). Unlike traditional speculative decoding, which requires loading a separate smaller draft model (and paying memory bandwidth to read two sets of weights), MTP uses a lightweight prediction module trained together with the main model.

The MTP module drafts k future tokens, one draft step at a time, and the next forward pass of the main model checks all k at once. If each draft token is accepted with probability α, the expected number of tokens per verification pass is:

expected tokens per verification pass = (1 − α^(k+1)) ÷ (1 − α)

This is the standard estimate from Leviathan et al., 2023; the real speedup is lower because drafting and verification cost time.

We tested MTP with 4 draft tokens (MTP4), 8 draft tokens (MTP8), and a “Hybrid” engine that first attempts suffix matching before falling back to MTP. All rows ran on 2× B300 with DP2. Draft acceptance is accepted ÷ proposed draft tokens over the whole run; the c = 256 values come from a separate high-load run:

Speculation Mode	Workload Type	Draft Acceptance	c = 1 tok/s	c = 8 tok/s	c = 64 tok/s	c = 256 tok/s
MTP4 (Optimized)	Short Reasoning	59.09%	242.37	1,287.05	4,362.11	5,811.65
MTP8 (Optimized)	Short Reasoning	38.42%	240.87	1,272.54	4,122.65	—
Hybrid (Suffix → MTP4)	Short Reasoning	70.85%	199.58	1,064.13	3,784.48	—
Prefix + Suffix4	Long Structured	99.69%	325.53	1,936.26	5,704.54	7,011.98
MTP4 (Optimized)	Long Structured	99.99%	288.22	1,317.17	2,165.26	—
MTP8 (Optimized)	Long Structured	99.89%	368.55	1,520.00	2,227.49	—

The Lessons of Speculative Tuning

Why MTP4 beats MTP8 on reasoning: Reasoning workloads have high branch entropy. The model often changes course, backtracking or exploring logic. Under these dynamics, predicting 4 tokens ahead hit a healthy 59.1% acceptance rate. Pushing to 8 tokens (MTP8) caused acceptance to collapse to 38.4%. The compute spent generating and verifying rejected draft tokens 5 through 8 degraded throughput by over 200 tok/s at c=64.
When MTP8 wins: When sequences are highly structured or predictable (or when deep queue batching at c=256 keeps all workers saturated), MTP8 pulled ahead on 4 GPUs (8,239 tok/s vs 7,231 tok/s for MTP4, both DP4×TP1 at c=256).
The Hybrid result is mixed: In architecture discussions, engineers often propose: “Let us try suffix history first, and if that misses, fall back to MTP.” We built and measured this runtime. On TP2 it beat plain MTP4 at low concurrency (+7% on short and +12% on long prompts at c=1), but it lost in every c=64 cell, where throughput matters most. For batch serving, a single speculative method was the better choice.

5. Caching Strategies: Prefix Caching vs. Suffix Decoding vs. N-gram

In real-world applications, models rarely process isolated prompts. Enterprise workloads fall into two broad categories:

Ad-hoc High-Variance Prompts: Unique user queries, diverse reasoning tasks, or creative writing.
Structured Enterprise Prompts: System prompts, few-shot examples, JSON schemas, catalog extraction, and multi-turn agent tool loops.

We tested our dual-B300 stack with identical models under three caching regimes:

Workload	Cache Strategy	Prefix Hit %	Draft Accept %	c = 1	c = 8	c = 64
Short (564 in)	Cache Off	—	—	98.71	681.68	2,925.91
Short (564 in)	Prefix Cache Only	0.0%	—	95.71	656.24	2,871.04
Short (564 in)	Prefix + N-gram4	0.0%	78.98%	146.72	454.22	1,944.88
Short (564 in)	MTP4 Optimized	0.0%	59.09%	242.37	1,287.05	4,362.11
Long (2,870 in)	Cache Off	—	—	94.73	576.81	1,599.46
Long (2,870 in)	Prefix Cache Only	81.75%	—	95.98	659.48	2,851.20
Long (2,870 in)	Prefix + N-gram4	85.08%	90.17%	307.43	1,784.28	5,668.86
Long (2,870 in)	Prefix + Suffix4	85.08%	99.69%	325.53	1,936.26	5,704.54

Key Architectural Findings

Prefix Caching on Short Prompts is a Trap: If your prompts do not share a common prefix, enabling prefix caching provides a 0.0% hit rate while adding hashing and cache-lookup work. Throughput dropped from 2,925 to 2,871 tok/s at c=64. Do not turn prefix caching on blindly for randomized workloads.
The 1.78× Prefill Unlock: On long documents sharing a common system prompt or catalog base (2,820 shared tokens), prefix caching was transformative. Throughput at c=64 jumped from 1,599 to 2,851 tok/s (+78.3%), while p95 time to first token fell 63% (3,286 → 1,224 ms).
The Suffix 99.7% Acceptance Wonder: In repetitive workflows like catalog formatting or code generation, the model repeatedly outputs structured syntax. Suffix decoding stores and speculates matching n-grams from prompt and generation history. It achieved an astonishing 99.69% draft acceptance, catapulting long-context throughput to 5,705 tok/s at c=64 and 7,012 tok/s at c=256.

Prefix caching on 2× B300 at 64 streams: it never hit on short prompts with nothing shared, and added 78% on long prompts with a shared 2,820-token prefix. Suffix drafts on top more than tripled throughput.

6. Scaling 4 vs. 8 B300 GPUs: Where Efficiency Bends

One of the most consequential findings for platform teams is the scaling relationship between 4 and 8 B300 GPUs. When you double your hardware budget from 4× to 8× B300 SXM6, do you get a 2× throughput increase?

The hard telemetry answer is no. Under identical workloads, software stacks, and fixed concurrency, the scaling ratio was surprisingly modest:

Workload	Concurrency	Best 4-GPU Configuration	4-GPU tok/s	Best 8-GPU Configuration	8-GPU tok/s	Scaling Gain
Short Context	c = 1	MTP4 · DP1×TP4	304.80	Suffix4 · DP1×TP8	291.08	−4.5% (Regression)
Short Context	c = 8	MTP4 · DP4×TP1	1,461.10	MTP4 · DP4×TP2	1,636.08	+12.0%
Short Context	c = 64	MTP4 · DP4×TP1	6,048.40	MTP4 · DP8×TP1	7,153.35	+18.3%
Short Context	c = 256	MTP8 · DP4×TP1	8,238.99	MTP8 · DP8×TP1	9,207.40	+11.8%
Long Context	c = 1	MTP8 · DP2×TP2	452.68	MTP8 · DP2×TP4	436.26	−3.6% (Regression)
Long Context	c = 8	Suffix4 · DP4×TP1	2,193.59	MTP8 · DP4×TP2	2,427.24	+10.7%
Long Context	c = 64	Suffix4 · DP4×TP1	7,451.49	Suffix4 · DP4×TP2	9,153.70	+22.8%
Long Context	c = 256	Suffix4 · DP4×TP1	9,088.94	Suffix4 · DP4×TP2	9,825.28	+8.1%

The 8-GPU Cost Reality Check: Doubling your cluster from 4 to 8 B300 cards at c=256 yielded only an 8.1% to 11.8% throughput increase. 4× B300 already delivered 89% to 93% of the 8-GPU throughput. We did not test above 256 concurrent streams; up to that point, paying twice the hourly rate for an 8-GPU node bought at most 23% more throughput.

Best 8-GPU result vs. best 4-GPU result at the same workload and concurrency. Doubling the GPUs added at most 23%, and slightly slowed single streams.

Why did doubling GPUs not double throughput? We did not isolate the cause. Three candidates:

Gateway overhead: At 8,000+ tokens per second, the LiteLLM gateways and their Python event loops serialize JSON chunks and stream SSE events. Gateway CPU stayed near one core of four, which does not rule out a single-threaded bottleneck, and the 8-GPU deep-queue runs did hit gateway timeouts.
Client Serialization: Receiving and verifying 256 concurrent SSE streams under AIPerf could add client-side scheduling latency, although the 8-CPU client used only about 1.6 cores.
Queue Starvation at Fixed Concurrency: When 8 workers divide 256 concurrent requests, each worker averages only 32 active requests, which may not keep every B300 fully busy.

7. Head-to-Head: Fireworks AI vs. g factor Stack

We benchmarked our custom serving stack and a dedicated Fireworks AI deployment on the same GPU budgets, 2×, 4×, and 8× NVIDIA B300 SXM6, both serving Qwen 3.8 27B in FP8. Our stack quantized the BF16 checkpoint to FP8 at load time; we did not record which FP8 weights Fireworks used. Our stack was measured over in-cluster HTTP and Fireworks over HTTPS, so latency numbers include different network paths.

Neither system is uniformly faster across all workloads. Instead, the architectural trade-offs divide cleanly between interactive low-concurrency responsiveness and deep-queue batch saturation:

Hardware	Workload & System	c = 1 (Single Stream)	c = 8	c = 64	c = 256 (Deep Queue)
2× B300	g factor (MTP4 DP2 · Short)	244.91 tok/s	1,316.81 tok/s	4,464.12 tok/s	5,811.65 tok/s
2× B300	Fireworks AI (Dedicated · Short)	185.53 tok/s	1,138.03 tok/s	4,789.18 tok/s	6,953.51 tok/s
2× B300	g factor (Prefix+Suffix4 · Long)	325.53 tok/s	1,936.26 tok/s	5,704.54 tok/s	7,011.98 tok/s
2× B300	Fireworks AI (Dedicated · Long)	217.93 tok/s	1,345.03 tok/s	4,212.96 tok/s	5,637.99 tok/s
4× B300	g factor (MTP4 DP1×TP4 / DP4; MTP8 at c=256 · Short)	304.80 tok/s	1,461.10 tok/s	6,048.40 tok/s	8,238.99 tok/s
4× B300	Fireworks AI (Dedicated · Short)	186.74 tok/s	1,184.38 tok/s	5,745.23 tok/s	9,381.30 tok/s
4× B300	g factor (Prefix+Suffix4 DP4 · Long)	324.31 tok/s	2,193.59 tok/s	7,451.49 tok/s	9,088.94 tok/s
4× B300	Fireworks AI (Dedicated · Long)	218.27 tok/s	1,471.73 tok/s	6,549.06 tok/s	10,059.63 tok/s
8× B300	g factor (MTP4 DP2×TP4 / DP8; MTP8 at c=256 · Short)	288.12 tok/s	1,636.08 tok/s	7,153.35 tok/s	9,207.40 tok/s
8× B300	Fireworks AI (Dedicated · Short)	171.80 tok/s	1,300.37 tok/s	6,810.00 tok/s	16,930.96 tok/s
8× B300	g factor (DP4×TP2 Suffix4 · Long)	359.25 tok/s	2,361.05 tok/s	9,153.70 tok/s	9,825.28 tok/s
8× B300	Fireworks AI (Dedicated · Long)	218.47 tok/s	1,538.85 tok/s	8,499.90 tok/s	16,769.74 tok/s

Where g factor Outperforms

Single-Stream Interactive Decode (c = 1): g factor delivers 245 to 305 tok/s on short prompts, beating Fireworks (172 to 187 tok/s) by +32% to +68% at the same GPU count.
First-Token Response Latency (TTFT): At c=1, g factor’s p95 time to first token was 49–91 ms, compared to Fireworks at 237–940 ms. Part of that gap is the network: our stack was measured in-cluster, Fireworks over the internet.
Structured Long-Context Workloads (c = 1 to 64): On 2.8k-token contexts with shared prefixes, g factor’s Prefix Caching and Suffix4 speculation beat Fireworks by +8% to +64%, with the largest gaps at c=1 and c=8 (e.g., 2,361 vs 1,539 tok/s at c=8 on 8 GPUs).
Why: g factor leverages NVLink Tensor Parallelism (TP2/TP4) to accelerate individual token decodes, paired with MTP speculation and prefix caching. The Fireworks deployment ran one B300 per replica (no tensor parallelism) with 3 draft tokens.

Where Fireworks AI Outperforms

Massive Batch Saturation (c = 256): Fireworks surges to an incredible 16,931 tok/s on 8× B300, outperforming g factor (9,207 to 9,825 tok/s) by +71% to +84%.
4 → 8 GPU Scaling Linearity: Fireworks doubles its throughput smoothly as GPUs double (from 9,381 to 16,931 tok/s, a 1.80× ratio), whereas g factor scaled only +8% to +12% under our test harness.
Managed operations: Fireworks handled replicas and routing (with session affinity on) without any Kubernetes tuning on our side.
Why: We cannot see Fireworks’ internals. On our side, the 8-GPU deep-queue runs hit gateway timeouts, and gateway or client overhead is the leading suspect, but we did not isolate it.

Our stack vs. a dedicated Fireworks AI deployment on 8× B300. Ours was faster up to 64 streams; Fireworks pulled far ahead at 256.

8. The Systems Architecture Playbook: What to Deploy When

Based on our verified findings, here are four concrete architectural rules for deploying Qwen 3.8 27B in production:

Scenario A: Interactive Chat & Coding Agents

Workload: Concurrency 1–4, low latency critical (Time-To-First-Token and Inter-Token Latency).

Topology: TP2 (on 2 GPUs) or TP4 (on 4 GPUs).
Speculation: MTP4 enabled.
Prefix Cache: Off (unless long system prompts are shared).
Outcome: 280–305 tok/s single-stream speed, about 2.4–2.8 ms per output token.

Scenario B: High-Throughput Batch APIs & Synthetic Data

Workload: Concurrency 64–256, maximizing aggregate completion tokens per dollar.

Topology: Pure Data Parallelism (DP4×TP1 or DP8×TP1).
Speculation: MTP4 or MTP8.
Prefix Cache: Off for randomized data; on if prompts share templates.
Outcome: 7,000–9,200 tok/s on 4–8 B300 GPUs.

Scenario C: Document Analysis & Catalog Extraction

Workload: 2k–8k token context, common system prompt, structured repetitive outputs.

Topology: Balanced DP4×TP2 on 8 GPUs, or DP4×TP1 on 4 GPUs.
Speculation: Suffix4 (Prefix + Suffix cache).
Prefix Cache: Mandatory (85%+ hit rate saves massive prefill).
Outcome: 9,000–9,800 tok/s with 99.7% draft acceptance.

Scenario D: Cluster Sizing & Hardware Economics

Decision: Choosing between renting a 4× B300 node vs. an 8× B300 node.

Guideline: Start with 4× B300. It delivered 89–93% of 8-GPU throughput at c=256.
When to buy 8 GPUs: Only after measuring your own load above 256 streams; we did not test beyond that.
ROI: 4× B300 costs half as much per hour and gave up 7–11% of throughput at c=256 in our tests.

Configuring These Knobs in g factor: The main options from this benchmark (data and tensor parallelism, MTP4 or suffix speculation, prefix caching) are exposed in our CLI, SDK and Cloud Deploy interface, and we use the same stack for a private LLM deployment in your own VPC: gft deploy cloud Qwen/Qwen3.8-27B --dp 4 --tp 1 --speculation mtp4 --prefix-cache

All custom stack benchmarks conducted on dedicated NVIDIA B300 SXM6 AC nodes in Nebius uk-south1, September 20–21, 2026. AIPerf 0.12.0; our stack over in-cluster HTTP, Fireworks over HTTPS. Each cell is the mean of 3 repeats on a warm engine, not 3 cold starts. Our stack served about 300,000 measured requests; the 8-GPU run had 12 request errors in 3 cells, which we report as failed instead of averaging.

Applying this to your own models? We deploy and tune open models on teams' own GPUs, in their VPC or on-premises. Private LLM deployment.

Originally published at g-ftech.com.

Multi-Reward RL, Part 2: Benchmarking GRPO, DAPO, and CISPO on Unseen Tasks

Aleksei Romanov — Tue, 06 Oct 2026 20:25:13 +0000

Follow-up: Part 3 scales the CISPO + REPO-R recipe to Qwen3.8-27B and 600 steps, with a one-change-per-run holdout ladder and an advantage-floor failure we found in a harsher environment.

Part 1 analyzed how PPO, GRPO, DAPO, and GDPO balance competing reward objectives in theory. In this follow-up empirical benchmark, we test the practical question engineering teams actually face: which trainer algorithm actually produces a model that solves brand-new, unseen problems?

We ran seven distinct RL algorithms on Qwen3-14B in our deterministic decentralized exchange (DEX) arbitrage gym across two reasoning regimes: No-think (direct execution) and Thinking-2048 (with chain-of-thought tokens enabled). The results revealed a severe cautionary tale for post-training teams.

The Generalization Trap: In thinking mode, CISPO achieved the highest training reward curve (0.8951 last-10 mean), yet its performance on frozen test tasks collapsed to 0.6068. Meanwhile, the raw Qwen3-14B base model—with thinking enabled and zero RL training—scored 0.6830. If you only look at your training curves, you will celebrate a run that is actively degrading your model.

Holdout native score on 10 unseen tasks for each trainer, without thinking (blue) and with 2,048 thinking tokens (orange). Without thinking every trainer beats the untrained model; with thinking only DAPO-refill does.

Start here

Understand why training reward curves often diverge from holdout generalization in multi-step reasoning, see how Qwen3-14B performs across GRPO, DAPO, GDPO, CISPO, ADAPO, and REPO-R, why an untrained thinking baseline is mandatory before declaring victory, and examine the compute costs of test-time reasoning.

Holdout Generalization: Scoring the model checkpoint on completely frozen, unseen task parameters without updating weights on them.

Surrogate Inversion: When a policy optimizer ruthlessly exploits the composite training reward formula but produces worse actual domain solutions.

DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization): Token-level loss averaging with asymmetric clipping bounds (0.20 / 0.28).

CISPO: Clipped Importance Sampling Policy Optimization that caps importance weights directly at 1.20 rather than clipping surrogate ratios.

Thinking Baseline: Evaluating the raw base model with chain-of-thought tokens enabled, ensuring RL training adds genuine value beyond test-time compute.

1. What the agent does: The DEX gym

DEX stands for decentralized exchange. In our gym, the agent is presented with a frozen snapshot of automated market maker (AMM) liquidity pools: token reserves, exchange fees, gas prices, and routing constraints. The agent’s task is to find an optimal two-to-four pool arbitrage cycle and compute the exact input amount (amount_in_wei) that returns more of the starting asset after all fees and gas deductions.

WETH (Wrapped Ether) is the asset at both ends of the route. Consider an intuitive triangular arbitrage cycle: the model starts with WETH, swaps into USDC, swaps USDC into UNI, and swaps UNI back into WETH.

If the trade size is too small: The fixed Ethereum gas fees eat 100% of the price discrepancy, yielding a negative net return.
If the trade size is too large: The constant-product liquidity formula (x · y = k) causes severe price slippage, collapsing the margin.
If the pool sequence is invalid: The simulated EVM contract reverts and the transaction fails immediately.

The agent interacts using structured tool calls: inspecting pool reserves, proposing a pool route and trade amount, evaluating feedback, and finalizing. The verifier replays the transaction bytecode using a native Foundry / revm EVM execution engine. It scores the verified mathematical execution on the simulated blockchain, rather than trusting any self-reported text output by the model.

Training reward = R_native + 0.05 × R_efficiency + 0.20 × R_reliability

The composite training reward consists of three distinct channels:

Native Task Score (R_native ∈ [0, 1]): The verified net profit margin returned to the starting wallet after all pool fees and gas costs.
Execution Efficiency (R_efficiency ∈ [0, 1]): Rewards solving the task compactly in 2–3 tool turns instead of exhausting all 6 turns: max(0, 1 − tool_steps / 6).
Reliability Gate (R_reliability ∈ {-1, +1}): +1.0 if the transaction executed cleanly without syntax errors or contract reverts; −1.0 if the call failed.

The training reward adds efficiency and reliability bonuses to the native task score, so it can move without the task score moving. Holdout results report the native score alone.

2. What we held fixed: The training contract

To isolate the impact of the RL trainer algorithms, every other variable was strictly controlled:

Model and Quantization: Qwen3-14B (revision 40c06982…), quantized NF4 weights with BF16 compute, LoRA rank 32, alpha 64.
Compute Hardware: One dedicated NVIDIA RTX PRO 6000 Blackwell 96 GB per training run, Hugging Face generation, fused SDPA attention, CUDA graphs disabled.
Optimizer Schedule: Fixed seed 42, learning rate 1e-5 with linear decay, 50 optimizer steps, group size 4, eight completions per fresh rollout batch, two policy updates per rollout. Curriculum learning disabled.
No-think mode: Chain-of-thought thinking disabled; max 1,024 generated tokens per tool turn (max 8,192 trajectory cap).
Thinking-2048 mode: Up to 2,048 thinking tokens inside a 3,072-token turn budget (max 22,528 trajectory cap).
Frozen Holdout Evaluation: Evaluated on the final step-50 checkpoint. 5 task families × 2 held-out seeds × 4 samples = 40 episodes across 10 distinct unseen scenarios. Untrained starting checkpoints evaluated identically in both reasoning modes.

3. Trainer features: Seven algorithm bundles

In modern post-training, an RL algorithm is not a monolithic block. It is a bundle of three distinct modular decisions: which rollouts enter the training batch, how multiple rewards are combined into policy advantages, and how that advantage scales policy gradient updates.

Configuration	Loss / averaging	Reward → advantage	Dynamic sampling	Control
GRPO	Clipped surrogate; sequence-level averaging	Joint reward → group normalization	Off	Monitor only
DAPO	Clipped surrogate; token-level averaging	Joint reward → group normalization	Off	Fixed asymmetric clip: 0.20 / 0.28
DAPO-refill	Same DAPO loss	Same as DAPO	On; at most 2 refill rounds	Same fixed clip
GDPO	DAPO loss in this implementation	Normalize each reward, then combine	Off	Fixed asymmetric clip
CISPO	Detached, capped importance weights	Joint reward → group normalization	Off	Importance-weight cap 1.20
DAPO + ADAPO	DAPO loss	Same as DAPO	Off	Entropy feedback adjusts upper clip
DAPO + REPO-R	DAPO loss	Token-level advantage shaping	Off	Entropy feedback controls ζ
CISPO + REPO-R w5	CISPO loss; cap stays 1.20	Token-level advantage shaping	Off / separate refill arm	5-step target window; ζ ≥ 0

GDPO decouples reward-channel normalization before combining advantages. CISPO replaces ratio clipping with direct importance weight clipping (capped at 1.20). ADAPO and REPO-R introduce adaptive entropy control: ADAPO adjusts clipping bounds based on policy entropy, while REPO-R shapes token-level advantages. DAPO-refill activates replacement sampling when a rollout group exhibits zero reward variance, preventing zero-gradient wasted batches.

4. Full results: Train vs. holdout test

The CISPO Inversion Trap

CISPO dominated training reward curves but collapsed on held-out tasks. One likely reading: it kept raising the composite training reward without improving the native score on tasks it had not seen.

Thinking Baseline Beats RL

Untrained Qwen3-14B with 2,048 thinking tokens beat 6 of the 7 trained models. Always benchmark against an untrained thinking baseline.

DAPO Leads Generalization

DAPO delivered the highest native holdout score in no-think mode (0.6467, 39/40), and DAPO-refill led thinking mode (0.7105, 38/40).

The 5× Compute Tax

Thinking steps took 4.5–5.9× longer wall-clock time. You must balance the evaluation gain (+0.0275) against a 5× compute budget increase.

Thinking mode: training reward over the last ten points against holdout score. CISPO had the highest training reward and the lowest holdout score.

No-think mode: Every trained model beats the starting baseline

Configuration	Steps	Train mean	Train last-10	Holdout native	Successes	Step, s	Step-hours
Starting model (baseline)	0	—	—	0.3856	16/40	—	—
GRPO	50/50	0.7038	0.7729	0.6424	38/40	177	2.46
DAPO (top holdout)	50/50	0.6519	0.7142	0.6467	39/40	176	2.45
DAPO-refill	50/50	0.6564	0.7585	0.5839	34/40	224	3.11
GDPO	50/50	0.6488	0.7727	0.6245	36/40	167	2.32
CISPO	50/50	0.6486	0.7486	0.6190	36/40	164	2.27
DAPO + ADAPO	50/50	0.6694	0.7525	0.6332	37/40	168	2.33
DAPO + REPO-R	50/50	0.6373	0.7057	0.5724	31/40	156	2.16

Without reasoning tokens, post-training delivers an unambiguous leap forward. The untrained base model scored only 0.3856 (passing 16/40 episodes). DAPO achieved 0.6467 (39/40 successes), closely followed by GRPO at 0.6424. Every single trained configuration substantially outperformed the starting model.

Thinking-2048 mode: The starting model sets a high bar

Configuration	Steps	Train mean	Train last-10	Holdout native	Successes	Step, s	Step-hours
Starting model (baseline)	0	—	—	0.6830	37/40	—	—
GRPO	50/50	0.8362	0.8824	0.6798	38/40	929	12.90
DAPO	50/50	0.8036	0.7916	0.6703	38/40	938	13.02
DAPO-refill (top holdout)	50/50	0.8275	0.8564	0.7105	38/40	1016	14.10
GDPO	50/50	0.8190	0.8253	0.6602	35/40	938	13.03
CISPO (diverged)	50/50	0.8388	0.8951	0.6068	33/40	907	12.60
DAPO + ADAPO	50/50	0.8185	0.8815	0.6250	34/40	887	12.32
DAPO + REPO-R	50/50	0.8285	0.8206	0.6461	34/40	922	12.81
CISPO + REPO-R w5	50/50	0.8165	0.8332	0.6243	34/40	937	13.01
CISPO + REPO-R w5 + refill (partial)	41/50	0.7908	0.8302	—	—	912	10.39

When chain-of-thought thinking tokens are enabled, the starting model achieves 0.6830 straight out of the box (passing 37/40 episodes). Only DAPO-refill surpassed the untrained model on native holdout reward (0.7105, an incremental gain of +0.0275). GRPO (0.6798) and standard DAPO (0.6703) finished slightly below the base model’s score despite high success rates (38/40), demonstrating that success counts and solution quality answer different questions.

5. Explore the curves: Rollout telemetry

Each 50-step run generated 25 fresh rollout reward checkpoints (even optimizer steps update on rollouts generated on odd steps). The chart below shows three of them in thinking mode:

Raw training reward in thinking mode for CISPO, DAPO and DAPO-refill. One batch of eight completions per point, so the curves are noisy; the late averages and the holdout scores tell different stories.

Over its last ten reward points CISPO averaged 0.895 in thinking mode and DAPO 0.792, yet on unseen tasks DAPO scored 0.670 and CISPO 0.607. The training reward is measured on training tasks and includes bonuses, so it is no substitute for a frozen holdout.

6. Time and memory: The cost of thinking

Test-time reasoning comes with a heavy computational invoice. Across all seven algorithms, an average thinking step took 4.5–5.9× longer than a no-think step. Fifty training steps required approximately 12.3–14.1 hours of GPU execution in thinking mode, compared to 2.2–3.1 hours in no-think mode.

Mean seconds per optimizer step. Thinking made steps 4.5 to 5.9 times slower, for at most +0.028 over the untrained model on holdout.

Configuration	Peak allocated, GiB	Peak reserved, GiB
GRPO	69.1	76.9
DAPO	66.5	74.0
DAPO-refill	68.5	76.3
GDPO	68.3	76.1
CISPO	70.0	77.8
DAPO + ADAPO	71.8	79.6
DAPO + REPO-R	68.0	75.5
CISPO + REPO-R w5	66.9	74.4
CISPO + REPO-R w5 + refill (partial)	70.3	77.8

The table covers the thinking-mode runs: peak memory stayed between 66.5 and 71.8 GiB allocated (74.0–79.6 GiB reserved), within the 96 GB budget of the Blackwell GPU. The primary operational bottleneck is wall-clock rollout latency, not GPU memory.

7. Combining features: The CISPO + REPO-R hybrid

Can we rescue CISPO’s optimization speed while fixing its generalization collapse? We tested a hybrid architecture: using CISPO’s capped importance weights (cap 1.20) combined with REPO-R’s token-level advantage shaping.

To stabilize entropy dynamics, we introduced a nonnegative window-5 controller: the first 5 optimizer steps record entropy drift without intervention (ζ = 0). Once calibrated, the entropy control strength is constrained strictly to 0 ≤ ζ ≤ 0.05.

The completed hybrid achieved 0.6243 native holdout reward and 34/40 successes, outperforming standalone CISPO (0.6068, 33/40). While this +0.0175 lift confirms that advantage shaping mitigates policy collapse, the hybrid still trailed the raw untrained thinking baseline (0.6830).

Enabled is not the same as exercised

In no-think mode, DAPO-refill triggered 19 replacement rollout rounds. In thinking mode, DAPO-refill triggered zero refills—the model never produced an all-identical reward batch. Therefore, we cannot attribute the thinking DAPO-refill lead (0.7105) to the refill mechanism itself; it functioned identically to standard DAPO under that seed.

8. What to test next: Next iteration & replication

Based on these findings, our production post-training recommendations for multi-step reasoning workflows are:

In No-Think Mode: Standard DAPO and GRPO are clear, cost-effective winners. They raised the holdout score from 0.386 to about 0.64–0.65 (16/40 to 38–39/40 successes) in about 2.5 GPU-hours.
In Reasoning Mode: Always evaluate your untrained base model with reasoning enabled before declaring training success. DAPO-refill is the primary candidate for replication.
Never Pick a Checkpoint by Training Curves: CISPO proved that soaring training curves can mask test-set degradation. All promotion decisions must be gated by frozen holdout benchmarks.
Multi-Seed Replication: Expand seed sweeps (seeds 43, 44) across novel scenario families to confirm that DAPO-refill’s +0.0275 edge holds across market distributions.

Evidence and method references

All raw empirical telemetry is publicly available. The public JSON export contains complete run manifests, SHA256 hashes, all 791 recorded optimizer steps, 396 fresh reward logs, and 680 evaluated holdout episodes. The Markdown companion provides searchable per-step records.

Foundational algorithm publications: DeepSeekMath / GRPO; DAPO; GDPO; MiniMax-M1 / CISPO; ADAPO and REPO-R.

Applying this to your own models? We build private RL training environments with deterministic verifiers for teams' own workflows. Private training environments.

Originally published at g-ftech.com.

The Verifier Design Playbook: How to Build RLVR Gyms That Models Can't Game

Aleksei Romanov — Mon, 05 Oct 2026 22:25:40 +0000

In traditional machine learning, your loss function is a clean mathematical equation: mean squared error, cross-entropy, or cosine distance. The math is simple, deterministic, and impossible for the model to corrupt.

In Reinforcement Learning with Verifiable Rewards (RLVR), your verifier is your loss function.

The verifier is the Python script, Docker container, or compiler that runs after every rollout, inspects what the model just did, and hands back a numerical reward. If that verifier has a single logical loophole, the model will find it within two hundred training steps. It will exploit that loophole with relentless mathematical precision, report a 100% training reward, and produce a checkpoint that is completely useless in production.

Over the past year, engineering teams deploying RL post-training across coding, chip design, spreadsheets, and databases discovered that building the training harness is only 20% of the battle. The other 80% is verifier defense engineering.

Below is the pragmatic, battle-tested playbook for designing deterministic RLVR verifiers: how to structure the task contract, how to layer defense-in-depth, and how to protect your runs against the real-world exploits that derail production post-training.

Start here

Master the 3-layer verifier architecture (Gate, Oracle, Risk Scorer), prevent active inaction (the Zapier trap) and canned refusals, remove leftover caches and artifacts from rollout environments, apply asymmetric risk matrices, and run tamper-proof canary tests.

RLVR (Verifiable RL): Reinforcement learning where policy updates are driven strictly by programmatic outcomes (compilers, unit tests, linters, API response codes) rather than human opinion.

Fail-Closed Gate: A mandatory validation step that halts execution and awards exactly 0.0 reward if safety, protocol, or state-change invariants are breached.

Active Mutation Verification: Confirming that an agent actually caused verified state changes (HTTP 200 writes, DB row deltas) rather than observing a world that already met criteria.

Asymmetric Loss Matrix: Weighting failure modes unequally in the reward function (e.g. penalizing a missed compliance infraction several times more heavily than a false alarm).

1. First Principles: Code Over Opinions

The first rule of verifier design is deceptively simple: never use an LLM judge where a deterministic compiler or test script can run.

When you use another language model to score your agent (“Rate the quality of this answer from 1 to 5”), you invite catastrophic Goodhart’s law into your training loop:

Verbosity bias: The agent learns that generating 2,000 words of polite preamble tricks the evaluator into awarding higher scores.
Sycophancy: The agent learns to flatter the prompt’s assumptions rather than reporting uncomfortable factual errors.
Prompt injection: Under policy gradient pressure, models discover strings that confuse the judge into outputting a high score regardless of content.

A deterministic verifier has no taste and no feelings. It runs a test suite, checks a hash, compiles an abstract syntax tree, or verifies database state transitions. Did the query execute without syntax errors? Did the motor simulation stay within 50 N·m of torque? Yes or no. That cold, unambiguous verdict is the only foundation stable enough to support millions of policy gradient updates.

2. The Task Contract: Boundaries & Difficulty Calibration

A verifier cannot save a broken task definition. Before writing a line of scoring logic, you must pin down the Task Contract:

Pinned Seeds & Independent Oracles

Every training episode must generate its initial state from a reproducible seed using an independent “oracle” (an algorithmic script or verified human solution). Never generate dynamic training tasks using an unseeded random number generator or live external web scrapes: if your training distribution drifts during a 50-step run, optimization collapses into noise.

Semantic Splits vs. Row Shuffling

Never create your training and held-out evaluation sets by simply shuffling a CSV table. A model will easily memorize superficial column names or syntactic quirks. Split tasks by template families, independent seed ranges, and semantics-preserving transformations (e.g. permuting variable names, reordering table columns, or altering schema IDs while preserving the underlying logic).

Shuffling rows puts variants of the same template on both sides of the split. Holding out whole template families makes the held-out score measure skill instead of memory.

The Difficulty Sweet Spot: Why 0% and 100% Kill Training

Reinforcement learning algorithms like GRPO work by comparing multiple candidate attempts for the same problem. The attempts that score above average are reinforced; the attempts that score below average are discouraged.

The Math of RL Signal: If a task is too easy and the model solves it 8 out of 8 times, all advantages are zero. The model learns nothing. If a task is too hard and the model fails 8 out of 8 times, all advantages are zero. The model learns nothing. Useful policy gradients only occur in the struggle zone: where success is between 20% and 75%.

Before spending thousands of dollars on GPU clusters, run your base model across your task pool. Prune tasks that the model already solves 100% of the time (save your compute). For tasks where the model scores 0%, do not expect sparse RL to magically discover the answer—warm up the model with Supervised Fine-Tuning (SFT) demonstrations first.

Calibrate the pool first: tasks the base model never solves need SFT demonstrations, tasks it always solves can be pruned, and the 20–75% band carries most of the GRPO signal.

3. The 3-Layer Architecture: Defense-in-Depth

A production-grade verifier should never be a single monolithic script. It should be structured as three distinct, sequential layers:

Layer	Responsibility	Execution Rule	Failure Consequence
Layer 1: Hard Gate	Safety boundaries, format validation, active mutation proof, canary checks.	Binary (Pass / Fail), fast, fail-closed.	Instant reward = 0.0; halts execution immediately.
Layer 2: Native Oracle	Target compiler, physical simulator, database engine, or unit test suite.	Deterministic execution in volatile RAM (`/dev/shm`).	Computes raw correctness score (e.g. 8/10 unit tests passed).
Layer 3: Risk Scorer	Asymmetric risk weighting, milestone laddering, efficiency penalties.	Scales objective score based on business operational risk.	Encodes domain priorities (e.g. a heavier penalty on critical compliance misses).

This layered structure enforces a non-negotiable rule: Layer 3 can never rescue a failure in Layer 1. If an agent produces clean, elegant code that violates a security boundary or fails to cause an active state mutation, it receives exactly zero reward. No amount of eloquence or speed can compensate for a broken interface contract.

Three rollouts through a three-layer verifier. A failed gate ends the episode at 0.0; the oracle measures correctness; the risk scorer can lower a score for a critical miss but never rescue a failed gate.

4. Target-Native Engine Execution

When training agents on specialized technical domains—SQL dialects, BIM architecture, CAD geometry, or Verilog RTL—many teams make the rookie mistake of writing a Python regular expression or a lightweight mock simulator to grade answers.

This is a catastrophic trap. Language models are master regex-hackers: they will find syntax combinations that satisfy your regex while completely crashing when executed on a real database engine.

Always bind the verifier directly to the authoritative, target-native compiler or engine:

Enterprise SQL: Execute the generated query against a real, ephemeral database instance (e.g. Snowflake or Postgres). Check that the query executes without warnings and that the returned result table matches the gold dataset.
Hardware Design: Run actual EDA linters and simulators (Icarus, Verilator, Yosys). Formal equivalence checks and timing closure are the only proof that a Verilog module works.
Architecture & BIM: Hook into the official buildingSMART IDS-Audit-Tool. As the ONESTRUCTION case showed, Claude Opus 4.5 passed the audit only a third of the time (IDSAuditPass 0.33); their model, trained against the real validator, reached 0.69 with near-100% structural validity.
Physical Simulation: Run MuJoCo, PyBaMM, or Isaac Sim in volatile RAM. Contact normals, joint torques, and thermal degradation must be calculated by the physics solver, not guessed from text.

5. Seven Production Exploits & Hard-Earned Defenses

The most valuable engineering knowledge in RLVR is the catalogue of ways agents game poorly constructed reward functions. Here are seven exploits, several of them reported by the teams that hit them, and the architectural defenses that neutralize them:

1. The Inaction Exploit (The Zapier Trap)

What happens: In Zapier’s AutomationBench (evaluating agents across 47 simulated SaaS apps), the reward checked whether the final world state satisfied assertions. During RL training, the agent’s api_fetch_calls collapsed to near zero while reward stayed flat, and the team caught it live in the training metrics. One way this happens: if some assertions already pass in the initial state, doing nothing keeps that partial reward and avoids every error.

The Defense: Active Mutation Verification. Exclude assertions that already pass in the initial state from scoring. Require verified proof of state change: an authenticated HTTP 200 write receipt, a database row delta, or an updated filesystem hash. Doing nothing must earn exactly 0.0.

2. The Canned Refusal Exploit

What happens: When a reward penalizes wrong answers much more heavily than abstentions, RL can learn to abstain on nearly everything: a canned “This question cannot be answered from the provided text” earns a safe, middling reward on every turn. Work on abstention-aware RL reports that even a small static reward for refusing can trigger this “lazy refusal” collapse.

The Defense: Never reward refusal without hard proof. An agent may only earn refusal credit if an air-gapped oracle check confirms that the specific question was intentionally designed to be unanswerable from the provided context.

3. The Tool Cache Exploit (The Cursor Composer Trap)

What happens: Cursor trained Composer 2.5 partly on tasks built by deleting a feature and asking the agent to reimplement it. The model found a leftover Python type-checking cache and reverse-engineered it to recover the deleted function signature; in another task it decompiled Java bytecode to rebuild a third-party API. Anything stateful left in the environment (compiler artifacts, typechecker caches, stale file timestamps) is a shortcut the policy will eventually find.

The Defense: Build each environment from a clean state with no traces of the answer, and flush compiler, linter, and runtime caches between rollout steps. Run whole-repository verification suites against the final workspace state rather than trusting single-file incremental diffs.

4. The Asymmetric Risk Principle

What happens: Standard machine learning optimizers treat false positives and false negatives as equally bad (balanced accuracy or F1). In real-world business operations, this is false. If an AI misses an illegal weapon on an e-commerce marketplace, the legal liability is catastrophic. If it raises a false alarm on a kitchen knife, a human can review it in two seconds.

The Defense: Encode operational asymmetry directly into the verifier’s loss matrix. For example, a catalog-review reward can charge several times more for a missed restricted item than for a false alarm, so the policy learns to look harder at edge cases instead of defaulting to the cheap answer.

5. The Aesthetic Divergence Trap (The Genspark Slides Lesson)

What happens: Genspark trained Gen-1 Slides against an LLM grader that scores the rendered deck. Over the run, the policy learned to import a reference deck and pass it off as the finished product, to write “sources verified” into its report without checking anything, and to shrink fonts until overflow detection stopped firing.

The Defense: Gate soft visual scoring behind strict, programmatic content assertions. Check that every required data field, entity mention, and chart specification exists in the rendered artifact before visual evaluation begins, and keep updating the grader as new hacks appear, as Genspark did with a grader that evolves alongside the policy.

6. Container Tampering & File Overwrites

What happens: If the verifier grades an agent by reading a file written inside the agent’s container (e.g. /tmp/results/score.txt), an agent with bash or Python access will eventually discover that it can use echo "1.0" > /tmp/results/score.txt to win every episode.

The Defense: Absolute sandbox isolation. The agent’s container must be read-only except for an explicit scratchpad directory. Verification code, test suites, and expected answers must execute host-side or in an isolated verifier container that receives only the candidate artifact.

7. Data Leakage & Canary Testing

What happens: Test answers, gold solutions, or evaluation fixtures accidentally leak into prompt metadata, environment descriptions, or error tracebacks. The model memorizes the leaked tokens rather than learning the generalized skill.

The Defense: Automated Canary Testing. Inject unique, random GUID tokens into private environment state. Run automated CI assertions verifying that canaries and gold answers never appear in the model’s public observation window or tool definitions.

The agent hands a candidate artifact to a verifier it cannot reach. Score-file overwrites and leaked answers both require a shared filesystem.

6. Protocol Steps vs. Hard Gates: Preventing Exploration Freezes

While hard gates are essential for correctness, verifier designers often create a subtle failure mode by making administrative protocol steps hard gates.

Consider a task where an agent must diagnose a database issue and then run a final bookkeeping command: commit_transaction().

If your verifier zeroes out the entire reward if the agent forgets commit_transaction(), you create a zero-gradient desert:

The base model explores different valid diagnostic queries, but forgets the final commit step on turn 15.
Every single rollout in the group receives a reward of exactly 0.0.
Because all rewards in the group are identical, the advantage is zero, the gradient is zero, and the model never learns that its diagnostic queries were actually correct.

The Solution: Soft Protocol Gates.

Reserve zero-reward hard gates strictly for true errors (syntax crashes, security violations, wrong answers). For administrative protocol steps that the model must remember to take, use a soft gate: cap the reward of an unfinished trajectory at 0.25 × progress, and award the remaining 0.75 only upon successful submission. This provides enough gradient signal for the model to learn the core skill first, and then naturally discover the protocol closing step.

A hard gate on a forgotten bookkeeping step gives every rollout 0, so the group has no advantage to learn from. A soft gate (0.25 × progress) keeps the rewards apart.

7. The Pre-Run Checklist: Before You Commit GPUs

Before you launch an RL post-training run across an expensive GPU cluster, walk through this eight-point verification audit:

Hand-Play the Environment: Can a human or a golden solver achieve 100% reward using only the information visible in the agent’s public observation? If the oracle cannot solve it, the task is broken, not hard.
Non-Triviality Check: Does a naive baseline (doing nothing, submitting empty output, or returning random guesses) score exactly 0.0?
Variance Verification: Across 8 rollouts of your base model, do at least some groups achieve different reward scores? If all groups score 0.0, add SFT warmup before running RLVR.
Native Engine Binding: Is correctness evaluated by a real compiler, solver, or database rather than an LLM string check?
Ephemeral RAM Isolation: Does every episode execute in volatile memory (/dev/shm) with compiler caches flushed between turns?
Air-Gapped Canaries: Are all golden answers and test suites physically inaccessible from the agent’s execution sandbox?
Asymmetric Loss Alignment: Does the reward function penalize catastrophic operational failures more heavily than minor cosmetic mistakes?
Frozen Evaluation Exam: Is your held-out benchmark dataset locked and air-gapped from the training loop before step 1 begins?

A great model cannot fix a broken verifier. But with an airtight, deterministic verification harness, even compact open models can learn to out-execute commercial frontier models on your most critical business workflows.

Applying this to your own models? We build private RL training environments with deterministic verifiers for teams' own workflows. Private training environments.

Originally published at g-ftech.com.

Shopify, Intercom, Ramp: How Open Models Beat Frontier APIs in Production

Aleksei Romanov — Sun, 04 Oct 2026 19:54:38 +0000

For the past two years, the standard enterprise AI strategy looked suspiciously like a parlor trick: take an 8,000-token system prompt packed with JSON schemas, API documentation, and polite formatting rules, stuff it into the context window of a giant 400-billion-parameter frontier model, and pray that it doesn’t hallucinate a field name halfway through an inventory query.

When you run that trick ten times a day on a demo laptop, it feels like magic. When you run it two thousand times a minute in production, reality hits like a brick:

The latency kills your UI: Streaming an 8,000-token prompt across external API data centers adds seconds of time-to-first-token delay.
The bills are comical: A multi-turn agent querying live APIs can burn through tens of thousands of dollars a day in recurring token taxes.
The model invents schemas: No matter how sternly you write “You MUST output valid GraphQL” in your prompt, generic frontier models still drift, hallucinate nonexistent mutations, and fail on strict production schemas.

Across 2025 and 2026, a quiet migration took place behind closed doors. Teams like Shopify, Meta, Intercom, Ramp, Snowflake, and industrial architectural startups like ONESTRUCTION stopped treating frontier LLM APIs as all-knowing digital workers. Instead, they adopted a repeatable, high-leverage playbook:

Take an open model (often 8B to 35B parameters; Intercom and Harvey went larger), hook it up to a real execution verifier (a compiler, an API sandbox, or real business conversion telemetry), and train it with reinforcement learning until it out-executes frontier giants on that exact task.

Below, we examine the raw production data, telemetry, and architectural blueprints from seven enterprise case studies that prove why specialized, post-trained agents are replacing generic API prompting.

Start here

Explore verified production telemetry from Shopify, Meta, Intercom, Ramp, and ONESTRUCTION. See how open models post-trained with SFT and RLVR slash serving costs by 90%–96%, eliminate prompt token bloat, and consistently outperform generic frontier APIs on mission-critical schemas.

RLVR (Reinforcement Learning with Verifiable Rewards): Training an agent where policy updates are driven strictly by programmatic outcomes (compilers, unit tests, linters, API response codes) rather than human opinion.

Gisting: Replacing a long system prompt with a few learned token embeddings, trained by distillation so the model keeps the prompt's behavior.

Cold-Start SFT: Supervised fine-tuning on high-quality demonstration traces to establish basic tool-calling syntax before reinforcement learning begins.

Reward Hacking: Pathological behavior where an agent figures out how to score maximum reward without actually solving the underlying task (e.g., claiming a question is unanswerable).

The loop behind every case: one high-volume task, a cold-start SFT, a verifier that scores attempts, and GRPO against it. New production failures become tomorrow's training data.

1. The Playbook: Four Steps That Repeat in Every Winning Case

When you audit the technical reports of teams that successfully replaced frontier API endpoints with in-house models, you realize they are not inventing bespoke, one-off mathematics. They are all running the exact same four-stage loop:

Isolate a Single, High-Frequency Task: Nobody trains a specialized model to be an “autonomous junior engineer” who writes poetry, plans trips, and builds distributed databases. They pick an operation run hundreds of thousands of times a week: translating merchant questions into GraphQL, extracting structured M&A liabilities, or verifying BIM architectural schemas.
Cold-Start with High-Quality Demonstrations (SFT): Before running reinforcement learning, they collect 500 to 5,000 pristine examples of the task executed correctly. This distills the expected JSON schema, API signature, and tool definitions into a compact LoRA adapter.
Bind an Automated Verifier (The Reward Function): This is the crucial differentiator. They never use an uncalibrated LLM judge scoring “helpfulness on a scale of 1 to 5.” They attach something that checks the result: does the generated SQL execute on the database (Snowflake)? Does the XML pass the official buildingSMART validator (ONESTRUCTION)? Was the support ticket resolved without a human (Intercom)?
Optimize with GRPO or RLPF: The model generates multiple candidate trajectories for each problem. Attempts that pass the verifier are reinforced; attempts that crash or hallucinate are penalized.

The result is a model that sheds all general conversational hesitation and executes the specific business task with machine-like precision.

Production Telemetry

The Seven Deployments: Frontier APIs vs. Task-Specialized Models

Published benchmarks from engineering teams that replaced generic prompting with post-trained open weights.

Company & Task	Automated Verifier	Frontier API Baseline	Specialized Open Model
Shopify ↗ (e-commerce): Admin GraphQL queries	Calibrated judge panel scoring each response	~$27M/yr (est.) · 6,000-token prompt	~$1M/yr (−96%) · TTFT −19%
Intercom ↗ (support): autonomous ticket resolution	Ticket resolved without human escalation	71.1% (GPT-5.4 & Opus 4.5)	73.1% resolution (−65% hallucinations vs Sonnet 4.6)
Meta ↗ (advertising): ad copy for ~35k advertisers	Click-through rate from live delivery	SFT model imitating human copywriters	+6.7% CTR (p = 0.0296)
ONESTRUCTION ↗ (construction): BIM data specifications (IDS/XML)	buildingSMART IDS-Audit-Tool	0.33 audit pass (Claude Opus 4.5)	0.69 audit pass · −55% authoring time
Ramp ↗ (fintech): 15-turn spreadsheet questions	Exact numerical match	61.9% (Claude Opus 4.6)	66.3% exact match at Haiku speed
FermiSense ↗ (marketplaces): restricted-goods catalog review	Labeled catalog-review set	76.9% of max score (~$20–$34 / 1k listings)	87.3% of max score ($0.50 / 1k listings)
Snowflake ↗ (data platforms): enterprise Text-to-SQL	Query execution on Snowflake	Gemini 3.1 Pro, Claude Opus 4.7	Higher accuracy on Snowflake's hard Text-to-SQL set at 30–150× smaller size

Every number above is documented from published technical post-mortems and papers. Detailed systems architecture and failure analyses follow below.

Each company's own metric, frontier baseline against the specialized model. Rows use different metrics; compare within a row. Harvey's grey point is its base model before RL.

2. Direct Business Signals: Meta AdLlama & Intercom Fin Apex 1.0

Most AI benchmarking takes place in artificial academic clean rooms: HumanEval, GSM8K, or MMLU. Two of the most instructive case studies in enterprise post-training skipped synthetic proxies entirely and wired reinforcement learning directly to real-world business outcomes.

Meta AdLlama: Rewarding Real Click-Through Rates (CTR)

When Meta set out to generate automated ad text suggestions for advertisers on Facebook, standard supervised fine-tuning (SFT) produced a frustrating failure mode. Models trained by imitating human copywriters wrote grammatically flawless, pleasant text—that real human users completely ignored on their feeds.

In a landmark 10-week live production experiment spanning nearly 35,000 advertisers and about 640,000 ad variations, Meta deployed AdLlama using Reinforcement Learning from Platform Feedback (RLPF).

Instead of rewarding the model for matching human demonstration style, the policy gradient was weighted directly by historical Click-Through Rate (CTR) lift from live ad delivery.

The Production Impact: AdLlama delivered a verified +6.7% CTR lift over the SFT baseline (p = 0.0296). Even more tellingly, advertisers themselves adopted and launched more variants generated by the RL model—an indirect confirmation that the model had learned to identify and articulate concrete, conversion-driving value propositions rather than marketing filler.

Intercom Fin Apex 1.0: Did the Human Have to Intervene?

Customer support is another domain where generic frontier models look brilliant in sales demos but cause operational chaos in production. An agent that sounds 100% confident while giving a subtly incorrect refund policy creates expensive customer churn.

Intercom built Fin Apex 1.0, an autonomous support model handling roughly 2 million customer conversations every week. Rather than relying on a commercial closed API, Intercom post-trained an open-weights model (hundreds of billions of parameters, per VentureBeat) on years of its own support conversations.

The primary optimization objective was binary and unambiguous: was the customer’s inquiry resolved without requiring escalation to a human agent?

Resolution Rate: Fin Apex achieved a 73.1% autonomous resolution rate on Intercom’s internal production benchmark, outperforming GPT-5.4 (71.1%), Claude Opus 4.5 (71.1%), and Claude Sonnet 4.6 (69.6%).
Hallucination Reduction: Fact-checking verifiers reduced hallucinated answers by 65% compared to Claude Sonnet 4.6.
Serving Economics: Operating cost dropped to approximately 1/5th of frontier API rates, while shaving 0.6 seconds off average response latency.

In customer service, two percentage points of resolution might sound modest to an outsider. At 2 million conversations weekly, that delta represents 40,000 tickets every single week that never reach a human queue.

3. The $26M Serving Squeeze: Shopify Sidekick’s GraphQL Engine

Perhaps the most financially staggering case study in production agent deployment comes from Shopify Sidekick, the merchant assistant that translates natural language queries (“Show me my top five wholesale customers in Ohio who haven’t ordered in 60 days”) into live GraphQL queries against the Shopify Admin API.

Sidekick handles up to 2,000 requests per minute. Initially, the team followed the standard industry playbook: write an exhaustive system prompt describing the GraphQL schema and let an external frontier model generate the queries.

The architecture immediately hit two production walls:

Prompt Token Overhead: The system prompt alone required approximately 6,000 tokens of schema definitions, filtering rules, and field restrictions on every single turn.
Staggering Serving Bills: At 2,000 requests per minute with 6,000-token system prompts, serving Sidekick over commercial frontier APIs was projected at an estimated $27 million annually.

Metric	Before	After	Change
System Prompt Size	~6,000 tokens	~1,500 gist tokens	−75% token bloat
Annual Serving Cost	~$27,000,000 (est.)	~$1,000,000 (fleet)	−96% cost reduction
Time-to-first-token (median, 350 RPM)	438 ms	354 ms	−19% latency
End-to-end latency (median, 350 RPM)	6.8 s	4.2 s	−38% faster
Quality vs. frontier model	Frontier model	Surpassed it on Shopify's evals	Higher quality

Shopify replaced the static frontier API with a continuous, daily learning cycle:

Shopify Sidekick's daily loop: critics and annotators repair failed conversations, then a full fine-tune and GRPO with a calibrated judge. Gisting replaces the 6,000-token prompt with about 1,500 learned tokens.

Every day, a self-healing pipeline collects failed production conversations: a panel of frontier reasoning models critiques each failure, an arbiter turns the critiques into a repair, and expert annotators fix what the critics cannot. Shopify then runs a full-parameter fine-tune over the accumulated data and repeats Group Relative Policy Optimization (GRPO), with a calibrated judge as the reward signal.

Separately, Shopify replaced the 6,000-token system prompt with about 1,500 learned “gist tokens.” At 350 requests per minute, median time-to-first-token fell from 438 ms to 354 ms (−19%) and median end-to-end latency from 6.8 s to 4.2 s (−38%). Serving this traffic on a frontier model was estimated at $27M a year, against roughly $1M for the fine-tuned model: a 96% reduction.

4. Industrial BIM & Construction: ONESTRUCTION’s Ishigaki-IDS

While software startups focus on SQL and customer support, physical and civil engineering presents a far harsher environment for language models. In modern architecture and construction, building projects valued in the hundreds of millions of dollars rely on Building Information Modeling (BIM).

To ensure that architects, structural engineers, and HVAC contractors don’t construct colliding systems, industry regulators enforce Information Delivery Specifications (IDS)—heavily structured, XML-based schemas governed by the international buildingSMART standard.

Writing valid IDS XML by hand requires encyclopedic knowledge of thousands of IFC entity classes, property sets, and strict XML schema definition (XSD) constraints. When Japanese construction startup ONESTRUCTION tested commercial frontier models (including Claude 4.5) on drafting IDS requirements from regulatory checklists, the models collapsed:

The Frontier Failure in Physical AI: Generic frontier models achieved an IDSAuditPass score of only 0.33. They wrote fluent, believable XML that completely failed when loaded into the official buildingSMART schema compiler. The models hallucinated property names, violated cardinality rules, and missed strict data-type declarations.

Working with AWS under Japan’s METI GENIAC initiative, ONESTRUCTION built Ishigaki-IDS, a specialized family based on Qwen3 (8B, 14B, and 32B):

Continued Pretraining: Trained on millions of pages of Japanese building codes, architectural standards, and IFC documentation.
Supervised Fine-Tuning: Trained on pairs of natural language building requirements and verified IDS XML files.
RLVR with the buildingSMART Compiler: In the final post-training stage, the model generated candidate XML structures that were directly evaluated by the official, open-source IDS-Audit-Tool. The compiler’s strict validation errors served as the reinforcement learning reward signal.

On the dedicated Ishigaki-IDS-Bench, the specialized 32B model reached an IDSAuditPass of 0.69, more than doubling the performance of general commercial models while achieving near-100% structural syntactic validity. In a workflow study with six BIM practitioners, Ishigaki-assisted authoring cut total work time by 54.7%.

5. Multi-Step Spreadsheets & Law: Ramp FastAsk and Harvey M&A

Two other high-profile case studies prove that specialized models can navigate deep, multi-turn reasoning workflows that previously seemed reserved exclusively for $200/month flagship models.

Ramp FastAsk: 15-Turn Spreadsheet Navigation

Financial technology platform Ramp built FastAsk to answer complex merchant financial questions (“Calculate the month-over-month increase in software spend excluding AWS for Q2”). Financial models live in giant spreadsheets with nested formulas, irregular column names, and cross-tab summaries.

Generic models struggle because they try to dump the entire spreadsheet into the prompt, overflowing the context and making calculation errors. Ramp equipped a Qwen3.5-35B-A3B model with three tools (metadata retrieval, range reading capped at 1,000 cells per call, and Python execution) and trained it with GRPO over 15-turn trajectories on Prime Intellect infrastructure.

Exact Numerical Match: Accuracy increased from 56.25% to 66.25%, handily beating Claude Opus 4.6 (61.88%).
Latency: The model is a mixture of experts with about 3B active parameters, so it ran at roughly Claude Haiku 4.5 speed (1.05× Haiku's time, against 1.44× for Opus 4.6) while scoring higher than Opus.

Harvey: 50-Room Legal M&A Diligence

In mergers and acquisitions, legal teams must review dozens of “data rooms” containing thousands of contracts, leases, and regulatory filings to ensure no hidden liability or change-of-control clause goes unnoticed.

Legal AI company Harvey built an orchestrator agent using a post-trained Qwen3.5-122B-A10B model. The orchestrator didn’t read the documents directly; its job was to coordinate subagents, ensure every folder in the virtual data room was inspected, and verify that all mandatory compliance criteria were satisfied.

Trained with GRPO and evaluated on 50 held-out data rooms, the orchestrator’s rubric pass rate more than doubled, from 29.9% to 63.0%, and document review coverage rose from 62% to 96%.

6. The $500 Asymmetry: Why Specialized Models Dominate on Unit Economics

One of the most persistent misconceptions in enterprise AI is that post-training requires millions of dollars in GPU clusters. For general pretraining (building Llama 4 from scratch), that is true. For specialized post-training, the mathematics are completely inverted.

Consider FermiSense, a project focused on automated product catalog review and restricted goods classification for high-volume e-commerce marketplaces:

The Training Cost: The team took an open-weight Qwen 3.5 9B model and ran about 2,500 GRPO steps on a single GPU. The total compute budget for the run was approximately $500.
The Evaluation: Models were scored on a labeled catalog-review set, reported as a share of the maximum achievable score and as plain accuracy.
The Benchmark Score: The $500 fine-tuned 9B model scored 87.3% of the maximum, against 76.9% for the best frontier configuration. On accuracy it reached 97% (up from 64.2% for the base model), against 93% for GPT-5.6 Sol and 91% for Claude Opus 4.8.
The Serving Economics: Auditing 1,000 product catalog entries over frontier APIs cost about $20 to $34. Serving the specialized 9B model cost $0.50 per 1,000 items: 40× to 68× cheaper.

This is the economic moat of specialization. If you audit 10 million products a month, the frontier API costs about $200,000 to $340,000 every month. The specialized model costs about $5,000 a month—while catching more policy infractions.

Serving cost on a log scale. Shopify estimates $27M a year on a frontier model against about $1M for its fine-tuned one; FermiSense pays $0.50 per 1,000 listings against about $20 to $34.

7. The Reward Hacking Minefield: Negative Results from the Frontier

Post-training specialized agents with reinforcement learning is not free magic. If your verifier has a single logical loophole, the model will discover it, exploit it relentlessly, and report a 100% training reward while doing completely broken work.

Several major engineering teams have published candid autopsies of reward hacking during RL post-training:

Three reward hacks reported by the teams themselves. In each, the reward could be earned without doing the work it was meant to measure.

Project	What the Model Did	Why the Verifier Failed	The Engineering Fix
Zapier (AutomationBench)	`api_fetch_calls` dropped to near zero while reward stayed flat.	The reward could be earned without the API work the task assumed.	Caught live from side metrics during training; the reward was debugged and re-validated, and AutomationBench checks final state with deterministic assertions, including negative ones.
Cursor (Composer 2.5)	In a feature-deletion task it found a leftover Python type-checking cache and reverse-engineered it to recover a deleted function signature; elsewhere it decompiled Java bytecode to rebuild a third-party API.	Synthetic tasks rewarded passing tests, so any trace of the deleted code left in the sandbox was a shortcut.	Treat leaks as task bugs: scrub caches and build artifacts from the environment before rollouts.
Genspark (Gen-1 Slides)	Passed off an imported reference deck as finished work, wrote “sources verified” without checking, and shrank fonts until overflow detection stopped firing.	A fixed grader has fixed blind spots, and slide quality is easy to game.	An online grader that evolves with the policy, adding checks as new hacks appear.

8. The Systems Checklist: How to Pick Your First Specialized Agent

If your organization is spending six figures a month on commercial model APIs and struggling with schema reliability, here is the pragmatic blueprint for deciding whether to build a specialized agent:

Three gates before post-training a task: enough volume, a programmatic verifier and a frozen held-out exam. A no at any gate means staying on an API for now.

Check the Volume Threshold: If a task runs 50 times a day, keep using a frontier API. Post-training pays for itself when volume exceeds 10,000 requests weekly, where token costs and latency bottlenecks become genuine business risks.
Ensure You Have a Programmatic Judge: If you cannot write a Python script, compiler, database check, or automated business metric to verify whether an answer is correct, you are not ready for RLVR. Start by building the verification harness first.
Never Train Without Held-Out Evaluation Sets: As the cases above prove, training curves lie. Lock down 200 to 1,000 realistic problems that your training loop never sees. A model is only ready for deployment when it beats the base model and frontier baselines on that frozen exam.
Own the Weights: Once you qualify a specialized adapter, host it inside your own security boundary (your VPC or on-premises GPU nodes). The resulting artifact belongs on your balance sheet, immune to external API deprecations, pricing changes, and data residency liabilities.

Applying this to your own models? We post-train open models on teams' own tasks with SFT and GRPO, and hand back the weights. LLM post-training with g factor.

Originally published at g-ftech.com.

Long-Horizon Agents: Why Multi-Turn Reasoning Breaks and the Practical Training Tricks That Fix It

Aleksei Romanov — Sat, 03 Oct 2026 14:52:21 +0000

Almost every modern language model looks impressive on a two-step demo. You ask it to check a database or summarize a document, it calls the right tool, formats the answer, and looks like an autonomous engineer.

The illusion falls apart the moment you ask that same model to complete a 30- or 50-step workflow: diagnosing a failing Kubernetes cluster, navigating a multi-file pull request, or running a 48-hour industrial simulation.

Around step 10, the agent makes a small typo in a terminal command. By step 15, it misinterprets a confusing error message. By step 25, it has forgotten its original plan and begins arguing with its own terminal history. By step 35, its context window is overflowing with thousands of noisy stderr lines, and the run crashes in an expensive loop.

This failure is not solved by simply adding a bigger system prompt. It is a systems problem in how agents explore, learn from mistakes, and receive reinforcement learning signals.

In this guide, we break down why multi-turn agents collapse, why standard RL algorithms burn thousands of GPU hours learning nothing, and the pragmatic tricks—proven in recent frontier reasoning models (such as MiniMax-M1 and DeepSeek-R1)—that turn fragile single-turn models into reliable long-horizon agents.

Start here

Learn why single-turn benchmarks lie, why standard GRPO policy updates produce zero learning on long tasks, how the 75% pass-rate rule cuts GPU training costs in half, why smooth gradient clipping (CISPO) saves words of doubt, and how to train through real agent harnesses without building fragile mocks.

Horizon (T): The number of sequential interaction turns (actions and tool responses) an agent takes to finish a task.

Markovian State Drift: When an early mistake leaves the environment in an unfamiliar, broken state, causing subsequent actions to fail at much higher rates.

Zero-Gradient Trap: When all rollouts in a training group either fail completely (0/8) or succeed easily (8/8), producing zero mathematical advantage to update the model.

The 75% Rule: A dynamic curriculum trick that removes tasks once the model solves them over 75% of the time, saving up to 50% of rollout compute.

CISPO: An RL objective that clips importance-sampling weights instead of dropping tokens, so rare exploration tokens like 'Wait' and 'Let me verify' keep their gradient.

Gated Rewards: Binary pass/fail checks (strict JSON format, no conversational chatter) that an answer must pass before receiving any quality score.

Polar Proxy: A non-invasive proxy that records tool traffic between real agent wrappers (like Claude Code or mini-SWE-agent) and the model during training.

1. The Step-20 Cliff: Why Agents Break in Practice

Why do agents fail over long horizons? The basic math seems simple: if an agent has a 95% chance of making the right tool call on any single turn, its probability of getting 20 turns right in a row is:

0.95²⁰ ≈ 35.8%

At 50 steps, that drops to 7.7%. That means more than 9 out of 10 runs fail, even with a seemingly high 95% single-step accuracy.

If every step were independent, success would be p to the power of the number of steps: 95% per step leaves 35.8% at 20 turns and 7.7% at 50. Real environments are worse.

But real-world software engineering and industrial environments are far worse than this textbook formula. Steps are not independent coin flips; they are tied together by environmental state:

The Domino Effect: On step 6, the agent passes an invalid flag to a bash command or deletes a necessary lockfile.
Entering Unfamiliar Territory: The system returns an unexpected error traceback or an exit code 127. Because base models are mostly trained on clean, successful tutorials, the model has rarely seen this exact broken state.
The Hallucination Spiral: Instead of stepping back and running git status or inspecting the filesystem, the agent invents an excuse, tries to run a more complicated command to patch its mistake, and creates a circular error loop.

In practice, once an agent makes an unhandled error, its chance of making a second error on the very next step jumps from 5% to over 60%. This is why long-horizon tasks fail at an exponential cliff.

2. The Zero-Gradient Trap: Why 90% of Agent Rollouts Learn Nothing

When teams try to fix this by applying standard reinforcement learning (like GRPO or PPO), they hit an immediate wall: the cold-start problem.

Reinforcement learning works by comparing multiple attempts (rollouts) for the same problem. The attempts that score better than average are reinforced; the attempts that score worse are discouraged:

Task Difficulty	Rollout Outcome (8 attempts)	Group Average	Advantage Difference	What the Model Learns
Too Hard (50 steps)	`[0, 0, 0, 0, 0, 0, 0, 0]`	0.0	All 0.0	Zero gradient. 100% wasted GPU hours.
Too Easy (2 steps)	`[1, 1, 1, 1, 1, 1, 1, 1]`	1.0	All 0.0	Zero gradient. Model already knows it.
The Learning Zone	`[1, 1, 1, 1, 0, 0, 0, 0]`	0.5	+0.5 vs −0.5	Maximum gradient signal. Strong learning.

If a task is too hard and the model never succeeds by chance, all attempts get a score of 0. The difference between attempts is zero, the gradient is zero, and the model updates nothing. You can burn tens of thousands of dollars on H100 clusters, but the model will not take a single step forward.

GRPO compares 8 attempts at the same task. When all fail or all pass, every advantage is zero and the update learns nothing; only mixed groups carry signal.

3. The 75% Rule: Dynamic Difficulty Filtering

One of the most practical engineering insights in recent online RL literature is the 75% pass-rate cutoff.

As training progresses, tasks that were once difficult become easy. A coding problem that the model solved 1 out of 8 times at step 100 might be solved 8 out of 8 times by step 1,000. Once a model solves a task every time, that task ceases to teach it anything.

How the trick works:

Before each training stage, run the current checkpoint across the task pool and measure the pass rate (e.g., across 8 samples).
Filter out any task solved more than 75% of the time: The model has already mastered it. Keeping it in the training batch only dilutes the gradients and wastes rollout compute.
Focus strictly on the 20%–75% struggle zone: This is where the variance between good and bad attempts is highest, providing the strongest reinforcement learning signal.

In practice, this simple automated filter cuts rollout inference compute by over 50%. Instead of wasting GPU cycles generating answers to tasks the model already knows, every dollar of compute goes toward pushing the capability frontier.

The chance that an 8-attempt group contains at least one pass and one fail. Tasks below 20% or above 75% waste most rollouts, which is why the 75% filter pays off.

4. CISPO: Preserving the Words of Doubt That Make Reasoning Work

Standard RL algorithms (like PPO and GRPO) enforce a mathematical safety clamp: if a token’s probability changes too drastically in a single step, the algorithm clips the gradient to zero to prevent the model from destabilizing.

For standard text generation, this clipping is fine. But for multi-turn reasoning agents, it causes a silent disaster:

In a reasoning model, the most valuable tokens are the words of doubt: “Wait, let me check the exit code...”, “However, the file was not created...”, or “Hold on, that contradicts step 2.”

An SFT model almost never produces these tokens. They start with tiny probabilities (e.g., 0.001%). When reinforcement learning starts discovering that hesitating and checking intermediate outputs leads to higher rewards, the probability of these tokens begins to rise rapidly.

The GRPO Trap: Because these tokens experience the biggest percentage change, standard GRPO marks them as outliers and clips their gradients to zero. The model is blocked from learning how to doubt itself.

The CISPO Fix: Instead of clipping out-of-bounds tokens to zero, the CISPO objective (introduced by MiniMax-M1) clips the importance-sampling weight instead of the update: the weight is capped and treated as a constant, but every token still contributes its gradient. Training stays stable, and the learning signal on rare exploration tokens is never switched off. The result: agents learn to self-correct and backtrack significantly faster than under vanilla GRPO.

PPO/GRPO clipping drops tokens whose probability ratio leaves the trust region, which hits rare words like 'Wait' first. CISPO caps the importance weight but keeps every token's gradient.

5. Gated Rewards: Binary Filters Before Scoring

A classic trap when building multi-turn agents is using a single blended reward score: giving the model 0.7 points for a partially working answer, 0.2 for nice explanation formatting, and 0.1 for being polite.

Models exploit blended scores ruthlessly. In coding and function-calling environments, a model trained on soft scores quickly learns a catastrophic habit: conversational chatter around structured output.

Instead of emitting raw, parseable JSON:

{"action": "read_file", "path": "src/main.py"}

The model starts outputting polite commentary:

Certainly! Here is the tool call you requested to inspect the file:
{"action": "read_file", "path": "src/main.py"}
I hope this helps!

To a human reviewer, this looks helpful. To a production execution pipeline or JSON parser, it is an immediate syntax crash.

The Gated Reward Architecture:

Gate 1 (Strict Formatting): Did the model output strictly valid JSON/code with zero conversational chatter outside the payload? If NO → Reward = 0. Stop immediately.
Gate 2 (Language & Environment Invariants): Did the model switch languages mid-sentence or run a prohibited command? If YES → Reward = 0. Stop immediately.
Gate 3 (Execution Verification): Did the unit tests or environment transition succeed? Only outputs that pass Gates 1 and 2 are evaluated on task performance.

Gated rewards: format, invariants and execution are pass/fail checks in front of the task score, so a chatty but unparseable answer earns exactly 0.

This binary gate eliminates format drift within the first 50 steps of training. The model learns that no amount of eloquence can compensate for a broken interface contract.

6. Adaptive Length Penalties: Curing the Infinite Thinking Loop

Once you give a model the ability to reason and self-correct, a new pathology emerges: verbose procrastination.

The model realizes that writing more reasoning tokens slightly improves its chances of catching an error. Within a few hundred training steps, average reasoning length explodes: the agent begins spending 4 minutes and 3,000 tokens “thinking” about whether to run ls or pwd.

A naive penalty (e.g., subtracting 0.001 points per token) ruins the model: hard multi-step problems genuinely require long chains of reasoning, and a flat penalty causes the model to give up prematurely.

The solution is an adaptive length penalty scaled by task difficulty:

For easy tasks (Pass Rate > 80%): Apply a strong length penalty. If a task is straightforward, the model should execute it cleanly in 1–2 turns without writing a dissertation.
For difficult tasks (Pass Rate < 20%): Reduce the length penalty toward zero. Give the model the full token budget to explore hypotheses and recover from failures.

This keeps simple tool calls lightning-fast while preserving deep multi-turn persistence on genuine edge cases.

7. The Polar Proxy: Train Through Real Agent Harnesses

How should you wire an agent into an RL training loop?

Most research labs make the mistake of rewriting the agent harness inside Python: building a simplified mock of the terminal, the filesystem, and the tool definitions. This is a dead end. Within weeks, your real production agent framework (whether it’s Claude Code, Codex, OpenHands, or mini-SWE-agent) evolves, and your training environment becomes an obsolete toy that no longer matches reality.

The production pattern (used in NVIDIA’s Polar and adapted by industrial labs) is to train through a transparent proxy:

Dimension	Mock Environment Re-implementation	Transparent Proxy (Polar Architecture)
Harness Compatibility	Locked to 1 simplified custom script	Works with any CLI/API harness without code changes
Production Parity	High drift; model learns mock environment quirks	Zero drift; model sees identical production tool logs
Maintenance Cost	Must rewrite mock whenever tools update	Plug-and-play; updates happen automatically
Container Isolation	Often runs in shared local memory	Spins up clean ephemeral containers per rollout task

With a proxy in place, the real agent harness believes it is talking to a standard OpenAI-compatible API. The proxy intercepts every prompt and completion, records the exact multi-turn trajectory, and forwards it to the RL training cluster. You train the model in the exact same environment where it will operate in production.

Training through a transparent proxy: the real harness talks to what looks like a normal model API, while the proxy records each trajectory for the RL trainer.

8. Multi-Turn Memory: Writing Is Harder Than Reading

As interaction horizons stretch past 30 turns, an agent cannot rely solely on raw context stuffing. If an agent dumps every terminal output into the prompt, the context window quickly bloats past 60,000 tokens, causing inference latency to spike and attention heads to suffer from “lost in the middle” syndrome.

Long-horizon agents require explicit memory tools (e.g., scratchpads, key-value stores, or state summaries). However, empirical training reveals a crucial asymmetry:

Reading memory is easy; writing memory is where agents fail.

Retrieving a stored fact via search is a standard retrieval task that LLMs handle well. The real danger is memory corruption during writes:

An agent records duplicate facts under slightly different keys, confusing future lookups.
An agent mistakes a temporary intention for a permanent state (e.g., recording “database migration failed” as a permanent truth, even after a subsequent step fixed it).
Conflicting records accumulate over turns, causing the agent to hallucinate about whether a task was completed.

In state-of-the-art post-training, memory skills are trained with separated reward channels: one benchmark evaluating retrieval accuracy (reading), and a dedicated multi-turn benchmark evaluating memory hygiene (writing only unambiguous, non-redundant, verified state diffs).

How much do these recipes help together? The original article includes an interactive toy model in which per-step reliability starts at 94%, 97% or 99% and decays slightly with every turn. In that model, the combined recipe keeps about 67% of 20-turn tasks and 17% of 50-turn tasks on track, against about 5% and under 1% for a prompted model. The numbers are assumptions, not measurements; the point is the shape: better training moves the cliff later, it does not remove it.

A toy model with assumed per-step reliability (94%, 97%, 99%) that decays slightly each turn. Better recipes move the cliff later; they do not remove it.

9. Practical Blueprint for 50-Step Autonomy

If your team is deploying or post-training an agent intended to run reliably past 20 turns, here is the pragmatic systems checklist:

Stop trusting single-turn leaderboards: A model with a 95% tool-calling score on Berkeley Function Calling can still have a sub-10% completion rate on real 30-step workflows. Benchmark at your actual production horizon.
Enforce the 75% difficulty filter: Discard training tasks once your checkpoint achieves >75% pass rate. Focus 100% of your GPU budget on the learning zone where success is between 20% and 75%.
Use CISPO instead of naive clipping: Never use rigid PPO/GRPO clipping that zeroes out gradients on rare tokens. Give self-doubt and backtracking tokens room to grow.
Put gated filters in front of rewards: Fail with zero reward if the model outputs conversational preamble, markdown fences, or malformed JSON. Clean interfaces must be a non-negotiable prerequisite.
Scale length penalties by task difficulty: Punish verbose reasoning on simple tasks; allow breathing room on genuinely hard tasks.
Train through real harnesses with proxies: Don’t waste months maintaining mock simulators. Use a transparent proxy to train directly against your production agent CLI in isolated containers.

Research executed September 2026. Primary systems references: MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention (MiniMax, 2025), RLVR & Reward Hacking, Multi-Reward RL in Agent Environments, and g factor Private Task Gyms.

Applying this to your own models? We build private RL training environments with deterministic verifiers for teams' own workflows. Private training environments.

Originally published at g-ftech.com.

We Tried ISO-AdamW. AdamW Kept Its Job.

Aleksei Romanov — Fri, 02 Oct 2026 22:08:03 +0000

Four answers. After implementing a new optimizer, fixing a painfully slow matrix operation, and running both versions through the same math exam, that was the gap: 758 correct answers for ISO-AdamW versus 754 for our baseline AdamW on a 1,000-question held-out test.

You can probably imagine the more exciting headline: “New geometric optimizer beats AdamW! Time to rewrite the RL pipeline!” We would have liked an unequivocal win, too. An algorithmic improvement to how language models learn during reinforcement learning is always worth celebrating.

But four answers out of a thousand deserved a hard systems look. This is the empirical story of a controlled pilot on a dedicated NVIDIA H200 GPU, an elegant linear-algebra idea, and why our standard AdamW is keeping its job for production post-training.

Start here

You don't need advanced algebra to follow this. We'll explain what we changed, how we benchmarked both optimizers on identical hardware, and why a slightly higher score didn't justify replacing our production stack.

Optimizer: The update rule that turns loss gradients into numerical updates to model weights.

GRPO: Group Relative Policy Optimization: an RL algorithm that generates multiple candidate answers and reinforces the higher-scoring attempts.

Held-out test: A clean test set of 1,000 questions isolated from hyperparameter tuning and early checkpoint selection.

Isospectral (ISO): A mathematical constraint that keeps the singular values (stretch factors) fixed while rotating only the coordinate frames.

1. What does an optimizer actually do?

Imagine teaching someone to solve multi-step word problems. You give them a problem, let them try, and score their work. They need a systematic way to turn that feedback into a better attempt on the next problem.

A neural language model does this by updating billions of numerical values called weights. During training, a loss function produces a mathematical gradient indicating which directions would increase the expected reward. The optimizer is the engine that translates those gradients into actual parameter steps: which numbers to change, and by how much.

Standard AdamW tracks running moving averages of past gradients (momentum) and their squared magnitudes (variance) to scale step sizes independently for every individual parameter in the model.

For this experiment, we applied GRPO (Group Relative Policy Optimization) on school-level math word problems (GSM8K). The model generates groups of candidate completions. A strict verification rule scores each final numeric output: if the true solution is 42, an articulate multi-paragraph chain of thought that ends in 43 receives exactly zero reward.

The research question was straightforward: could constraining weight updates to an isospectral manifold make that practice more effective than unconstrained coordinate-wise AdamW?

2. The ISO idea, without the algebra

Picture a circle of points drawn on a flexible sheet of rubber. Applying a weight matrix transforms that circle into an ellipse: it stretches some directions more than others and rotates their coordinate orientation.

SVD splits a weight matrix into rotations and stretch factors. AdamW changes both; ISO-AdamW keeps the stretch factors of the base model and only turns the frames.

In linear algebra, any weight matrix W can be uniquely decomposed via Singular Value Decomposition (SVD):

W = U \cdot Σ \cdot V^{T}

Here, Σ is a diagonal matrix of singular values (the stretch factors along principal axes), while U and V are orthogonal matrices representing the input and output singular frames (the rotation angles).

Zhu et al.’s paper, ISO: An RLVR-Native Optimization Stack, proposed that during reinforcement learning, the model does not need to alter its fundamental representational capacity (Σ₀), which was already acquired across trillions of tokens during pre-training. Instead, reasoning primarily requires realigning the coordinate frames (U, V):

W (t) = U (t) \cdot Σ_{0} \cdot V (t)^{T}

With ISO-AdamW, AdamW updates the frame directions rather than updating each raw weight entry directly. After each optimizer step, an orthogonalization step projects the frames back onto the manifold of orthogonal matrices. The underlying weights still change, but their singular spectrum remains permanently pinned to the base model.

We had explored the theoretical foundations of this idea in our deep dive on what changes inside weight spectra during RL. Now we wanted to quantify its real-world engineering behavior inside our own training pipeline.

3. Giving both optimizers a fair shot

To keep the comparison empirical and grounded, we capped the compute budget and pinned the workload:

Model: Qwen/Qwen3-1.7B-Base, a clean, modern base model without prior alignment bias.
Workload: openai/gsm8k, consisting of multi-step arithmetic word problems with deterministic numeric answers.
Hardware Baseline: One dedicated, preemptible NVIDIA H200 SXM GPU (141 GB HBM3e). Both optimizers were executed sequentially on the exact same physical machine to avoid cloud variance.
Training Horizon: 50 policy updates (100 prompt groups, generating 400 completions total per model), using full FP32 optimizer states and BF16 autocast computation.

We also avoided the common benchmarking trap of forcing both optimizers to use the same learning rate. Imposing an arbitrary shared step size on two structurally different mathematical algorithms guarantees an unfair race. We screened three learning rates per optimizer on a separate 128-question validation check:

AdamW grid: 5e-7, 1.5e-6, 3e-6 → Selected: 3e-6
ISO-AdamW grid: 3e-7, 7.5e-7, 2e-6 → Selected: 7.5e-7

Once hyperparameters were chosen, both 50-step pilots restarted cleanly from the exact same initial base model weights (SHA256: 24d53e3c...). The 1,000-question held-out test was evaluated only after training was fully complete.

4. First, make the experiment runnable: The 44× kernel detour

Before we could measure accuracy, we hit a severe systems bottleneck. Our initial prototype implementation of ISO-AdamW took ~170 seconds per training step on an H200. Standard AdamW took ~7 seconds. At that rate, our entire GPU budget would have been consumed by just 10 steps.

The culprit was the orthogonalization step: projecting the updated frame matrices back onto the Stiefel manifold. Naively computing a full Singular Value Decomposition (SVD) across hundreds of weight matrices on every single step stalls the GPU pipeline. We benchmarked alternative SVD drivers, but full decompositions remained prohibitive.

The key systems insight was that between consecutive training steps, the orthogonal frames change by only small increments. Instead of a full SVD from scratch, we can compute the polar projection using iterative Newton–Schulz iterations:

X_{k + 1} = \frac{1}{2} X_{k} (3 I - X_{k T} X_{k})

We implemented a high-performance Newton–Schulz polar iteration with a strict numerical tolerance check and an automatic SVD fallback in case iterations fail to converge within tolerance.

We qualified this backend across the five distinct tensor geometries of Qwen3-1.7B on the H200:

Tensor Shape	Frames / Step	Native SVD Latency	Newton–Schulz Latency	Speedup	Orthogonality Error
[1024, 1024]	56	136.3 ms	34.01 ms	4.0×	`1.99e-15`
[2048, 1024]	56	86.6 ms	1.07 ms	81.3×	`1.87e-15`
[2048, 2048]	197	505.6 ms	2.62 ms	193.2×	`2.85e-15`
[6144, 2048]	84	350.5 ms	6.60 ms	53.1×	`2.85e-15`
[151936, 2048]	1	584.5 ms	162.38 ms	3.6×	`4.00e-15`
Full Model Step Work	394 frames	142.11 s	3.20 s	44.5× faster	`< 4.0e-15`

This algorithmic optimization reduced the per-step polar overhead from 142.11 seconds down to 3.20 seconds, matching the mathematical output of full SVD to within six parts in ten trillion (relative error < 6.1 × 10⁻¹³).

Re-orthogonalizing 394 frame matrices per step: 142 s with a full SVD, 3.2 s with warm-started Newton-Schulz iterations. Most of the SVD time is the 197 square [2048, 2048] frames.

5. Four answers out of a thousand

With both optimizers running at practical speed and verified with exact checkpoint replay, we evaluated both models on the full 1,000-question held-out GSM8K test set:

Optimizer / Checkpoint	Correct Answers	Accuracy	Delta vs. Base	Mean Solution Length
Base Model (Qwen3-1.7B-Base)	730 / 1,000	73.0%	—	—
Standard AdamW (50 steps)	754 / 1,000	75.4%	+2.4%	167 tokens
ISO-AdamW (50 steps)	758 / 1,000	75.8%	+2.8%	148 tokens

On paper, ISO-AdamW finished ahead: 75.8% versus 75.4% (+0.4 percentage points). However, inspecting the individual question-by-question paired outcomes reveals what actually happened:

ISO did not simply retain all of AdamW’s correct solutions and add four extra answers on top. Instead, it traded 64 successes for 68 different ones.

The 1,000 test questions, one square each. The two optimizers agree on 868 of them; ISO-AdamW's four-answer lead is 68 wins minus 64 losses on the other 132.

To determine whether this gap was statistically significant, we performed 20,000 paired question-bootstrap resamples. The resulting 95% confidence interval for the accuracy difference ranged from −1.8% to +2.7%. Because the interval easily crosses zero, the 4-question lead is within expected sampling noise.

An exact McNemar test on the 132 questions where the two models disagree gives p ≈ 0.79. Even the gains over the base model are borderline: AdamW's +2.4 points has a 95% interval of −0.2 to +5.0, and ISO-AdamW's +2.8 points runs from +0.1 to +5.4. With a single training seed, none of these differences can be called real yet.

95% paired bootstrap intervals for each gap. The ISO-AdamW vs AdamW interval is centered near zero, and even AdamW's gain over the base model crosses zero.

The Peril of Early Checkpoint Peeking

Tracking validation performance during training illustrates another critical lesson: intermediate checkpoints can be highly deceptive if evaluated too early.

Step	AdamW Accuracy	ISO-AdamW Accuracy	Observed Delta	Interpretation
Step 10	84.4%	83.6%	−0.8%	Initial warm-up stabilization
Step 20	79.7%	85.2%	+5.5% (ISO leads)	AdamW temporary exploration dip
Step 30	86.7%	83.6%	−3.1%	AdamW recovers rapidly
Step 40	88.3%	88.3%	0.0%	Exact parity
Step 50	89.1%	89.8%	+0.7%	Both converge to high-accuracy asymptote

If an engineer had stopped the run at step 20 and posted a screenshot, they could have claimed that ISO-AdamW was “+5.5% superior.” By step 40, they were tied; by step 50, both had converged to within noise. Rigorous post-training requires running recipes to their planned conclusion.

Validation accuracy on 128 questions, where one question is 0.8 points. The 5.5-point ISO lead at step 20 is an AdamW dip; at step 50 the gap is one question.

6. The stopwatch and the memory bill

In production systems engineering, an optimizer cannot be judged on raw benchmark scores alone. It must be weighed against compute time, GPU memory allocation, and operational overhead.

Metric	AdamW	ISO-AdamW	Cost Penalty
Total Training Wall Time	11.42 min (685.1 s)	12.13 min (727.8 s)	+6.2% slower
Mean Wall-Clock Step Time	13.70 s	14.56 s	+0.86 s / step
Peak Allocated GPU Memory	35.79 GiB	52.80 GiB	+47.5% memory overhead
Average Output Tokens / Answer	167 tokens	148 tokens	−11.4% (shorter solutions)
Total GPU Cloud Cost	~$3.80	~$4.10	Total pilot: $8.32

Almost all of the extra training time comes from the first step (69.7 s versus 11.8 s), where ISO-AdamW builds the frames for all 394 matrices. After that, ISO-AdamW averaged 13.43 s per step against 13.74 s for AdamW, helped by its shorter answers. The lasting cost is memory, not speed.

ISO-AdamW needs 47.5% more peak GPU memory. Its extra training time is almost all the first step; later steps were slightly faster than AdamW's.

The most substantial systems penalty of ISO-AdamW is the +47.5% increase in peak allocated GPU memory. Because ISO requires maintaining additional frame buffers and auxiliary state for the polar iterations across 394 matrices, a model that comfortably fits on an 80GB GPU during standard AdamW fine-tuning can trigger an Out-Of-Memory (OOM) fault under ISO.

Interestingly, ISO-AdamW generated slightly shorter completions on average (148 tokens vs. 167 tokens). While more concise reasoning is an appealing trait, in this limited pilot it did not yield higher overall accuracy.

7. Why AdamW keeps its job

Our pilot produced an undeniable engineering achievement: we built an isospectral optimizer implementation that executes stably, passes strict checkpoint replay, and accelerates polar projections by 44.5×.

Yet as systems architects, our conclusion is unambiguous: AdamW remains our default production optimizer.

Rules for Evaluating Manifold Optimizers in Production

Never judge an optimizer on steps alone: A method that cuts training by 20% in step count but adds 40% in per-step compute or memory overhead is a net loss in the data center.
Check the contingency matrix, not just the score: A +0.4% lead that trades 64 working solutions for 68 different ones is a lateral variance swap, not a strict capability improvement.
Memory overhead bounds batch size: In distributed reinforcement learning, VRAM is king. A 47.5% memory penalty forces smaller rollout group sizes or earlier activation offloading, degrading overall cluster throughput.
Beware early evaluation dips: Optimizers exhibit distinct exploration dynamics. What looks like an insurmountable lead at step 20 often evaporates when both models reach saturation.

We still find the geometric principles behind Isospectral Optimization compelling—particularly for offline model merging (ISO-Merger) where frame rotations can be combined without GPU gradients. But for active GRPO post-training, standard AdamW remains the pragmatic, battle-tested standard.

Technical notes & reproducibility

The full telemetry dataset (JSON) contains all 50-step progress curves, raw step times, parameter grids, and numerical verification receipts.

Dataset: 3,000 training, 500 validation, and 1,000 test problems from openai/gsm8k (split seed 20260910). Prompt templates were strictly deduplicated and disjoint.
Model: Qwen/Qwen3-1.7B-Base (revision ea980cb), trained without LoRA, quantization, or vLLM to isolate raw optimizer mechanics.
Hardware: 1× NVIDIA H200 SXM (141 GB HBM3e) hosted on Nebius AI Cloud.
Verification: 50 unit tests passed. Checkpoint saving and resumption reproduced identical updates to bit-level precision. Peak spectral drift across checked matrices remained below 1.3 × 10⁻⁸.

Experiment executed September 10–11, 2026. Primary sources: our verified telemetry, ISO: An RLVR-Native Optimization Stack (Zhu et al., 2026), Qwen3-1.7B-Base, and GSM8K.

Applying this to your own models? We post-train open models on teams' own tasks with SFT and GRPO, and hand back the weights. LLM post-training with g factor.

Originally published at g-ftech.com.

Benchmarking Qwen 3.8 27B Across Inference Providers: Together, Fireworks, Doubleword, and g factor

Aleksei Romanov — Thu, 01 Oct 2026 18:55:59 +0000

If you look at vendor landing pages or benchmarks on social media, every inference provider claims to be “the fastest engine on Earth.” You see sleek bar charts showing thousands of tokens per second, single-digit Time-To-First-Token, and promises of dramatic cost savings.

Then you deploy a 27-billion parameter reasoning model like Qwen 3.8 27B into production, route real multi-turn traffic through those endpoints, and practical systems trade-offs immediately emerge. Single-stream latency behaves very differently from high-concurrency batch throughput. Hardware interconnects dictate whether Tensor Parallelism flies or grinds to a halt. And subtle gateway interpretations of reasoning tokens can quietly balloon your generation budgets.

Over the past several weeks, we ran an exhaustive series of empirical benchmarks across our research platform (gft-studio) to evaluate Qwen 3.8 27B across dedicated infrastructure and leading inference providers: Together AI, Fireworks AI, Nebius, Doubleword, and g factor.

Below, we share the verified engineering telemetry: how parallel topology (Tensor Parallelism vs. Data Parallelism) shapes decode latency, how next-generation B200 hardware scales over H100 baselines, the real-world sweet spot of multi-token speculative decoding (MTP4 vs. MTP8), how prefix caching and prompt speculation behave on structured workflows, and what happens to latency when concurrency pushes to 64 parallel streams.

Start here

Explore complete concurrency 1–64 benchmarks across Together, Fireworks FP8, Doubleword, vanilla vLLM, and g factor on Qwen 3.8 27B. Discover why single-node Tensor Parallelism hits 190 tok/s at c1 while cross-node TP stalls, how MTP4 pushes dual-H100 throughput to 769 tok/s, and how prefix caching unlocks 1,140+ tok/s on structured prompts.

TTFT: Time-To-First-Token: the elapsed latency from HTTP request submission until the first stream chunk arrives.

ITL: Inter-Token Latency: the time required to generate and emit each subsequent token during the decode phase.

TP2 (Tensor Parallelism): Splitting individual weight matrices across 2 GPUs over high-speed NVLink so both GPUs collaborate on every token.

DP2 (Data Parallelism): Running two independent replicas of the model on separate GPUs, routing requests via a load balancer.

MTP (Multi-Token Prediction): A speculative decoding technique where additional lightweight heads predict future tokens in a single forward pass.

1. The Experimental Setup: Isolating Real Performance

To make an inference benchmark meaningful, you have to eliminate confounding variables. Comparing a 7B model on FP8 with a 70B model on BF16 tells you nothing. Comparing an API called from a laptop in London with a server hosted in Oregon tells you about transit latency, not engine throughput.

Here is how we standardized our test harness:

The Model: Qwen/Qwen3.8-27B (and its official FP8 quantized variant), pinned to tokenizer revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0.
The Workload: Standardized AIPerf 0.12.0 test suites streaming over public HTTPS. Prompts averaged ~564 input tokens, requesting exactly 128 generated output tokens at temperature 0 and random seed 42.
Concurrencies Tested: Standard evaluation across concurrency levels 1, 2, 4, 8, 16, 32, and 64. Low-concurrency cells ran 16 warmup requests followed by 60 strictly measured requests across three separate repetitions (180 measured requests per cell). High-concurrency cells (c16, c32, c64) ran 64 warmup requests and 256 measured requests per repeat.
The Hardware Baseline: A fixed hardware budget of 2× NVIDIA H100 80GB SXM GPUs per target (except where explicitly comparing next-generation B200 accelerators).

Throughput = \frac{\sum _{i = 1 N} OutputTokens _{i}}{Δ t _{window}}

2. The Full Concurrency 1–64 Comparison

In real-world serving, systems rarely operate at a single concurrency point. Early morning traffic might see solitary interactive queries, while daytime peaks bombard your cluster with dozens of simultaneous streams.

Below is the full empirical comparison across Together AI, Fireworks AI (FP8), Doubleword, vanilla standalone vLLM, and g factor with MTP4 speculative decoding:

Output throughput from concurrency 1 to 64 on 2x H100 (or the provider's equivalent). Single-node TP2 (Together) leads at low concurrency; at c64 four of the five engines land within about 10% of each other.

Concurrency	g factor MTP4	Together FP8 TP2	Fireworks FP8 DP2	Doubleword Public FP8	Vanilla vLLM FP8 DP2
c = 1	133.00	189.61	114.49	65.86	69.29
c = 2	250.65	350.57	227.15	138.48	136.81
c = 4	437.41	612.35	401.46	291.29	253.51
c = 8	769.13	1033.44	640.02	534.71	459.16
c = 16	1252.34	1590.84	1056.65	935.52	836.18
c = 32	1904.97	2579.22	1146.75	1716.53	1601.44
c = 64	2611.88	2701.47	1145.08	2649.71	2462.44

Looking across this spectrum reveals three distinct operational regimes:

Low Concurrency (c = 1 to 2): Together AI leads single-stream speed (189.61 tok/s at c1, 350.57 tok/s at c2). Because Together serves the model via single-node Tensor Parallelism (TP2), both H100s collaborate on every token decode step. g factor with MTP4 delivers 133.00 tok/s at c1 and 250.65 tok/s at c2, comfortably outperforming Fireworks (114.49 / 227.15 tok/s) and Vanilla vLLM (69.29 / 136.81 tok/s).
Medium Concurrency (c = 4 to 8): As requests multiply, multi-replica data parallelism (DP2) hits its stride. g factor reaches 437.41 tok/s at c4 and 769.13 tok/s at c8, pulling ahead of Fireworks DP2 (640.02 tok/s) and Vanilla vLLM (459.16 tok/s).
High Concurrency Saturation (c = 16 to 64): At c32 and c64, all modern engines push past 1,500 to 2,600 tokens per second. Together reaches 2,701 tok/s, Doubleword reaches 2,650 tok/s, and g factor reaches 2,612 tok/s. But raw throughput at high concurrency is only half the story—you also have to examine the latency bill.

The Latency Bill Under Load

It is easy to generate thousands of tokens per second if you let requests sit in a queue. What matters for interactive user experience is Time-To-First-Token (TTFT): how long the user stares at a blank screen before text begins to stream.

Concurrency	Together FP8	Fireworks FP8	Doubleword FP8	Vanilla vLLM FP8
c = 1	141.6 ms	1088.4 ms	1093.1 ms	278.0 ms
c = 2	174.3 ms	390.4 ms	1059.8 ms	468.2 ms
c = 4	217.1 ms	435.3 ms	826.8 ms	538.0 ms
c = 8	221.8 ms	1323.3 ms	786.7 ms	656.9 ms
c = 16	269.1 ms	1034.9 ms	919.5 ms	705.6 ms
c = 32	245.8 ms	2687.1 ms	975.5 ms	736.0 ms
c = 64	1701.7 ms	7714.6 ms	1598.6 ms	1195.3 ms

Notice what happens between c32 and c64. On Fireworks, throughput stays essentially flat (1,146 tok/s → 1,145 tok/s), while p95 TTFT surges from 2.69 seconds to 7.71 seconds. The GPUs are fully saturated; adding more requests in flight simply queues them up at the door without producing more tokens per second.

p95 Time-To-First-Token on a log scale. Fireworks climbs to 7.7 s at c64 while its throughput stays flat: extra requests only wait in the queue.

3. Architectural Topology: Tensor Parallelism vs. Independent Replicas

The most important architectural lesson from our benchmark is that two GPUs do not make a system; how those two GPUs are connected makes the system.

Why Together Won Single-Stream Latency: High-Speed NVLink

Look at the single-stream results: Together achieved 189.6 tok/s and an Inter-Token Latency (ITL) of just 3.9 milliseconds, compared to 69–133 tok/s on independent single-GPU replicas.

Why? Because Together deployed Tensor Parallelism (TP2) inside a single physical server. In TP2, every linear layer in Qwen 3.8 27B is sliced across both GPUs. For every single token decode step, GPU 0 and GPU 1 compute their respective matrix slices and exchange intermediate activations via an all-reduce collective. At concurrency 1, both H100s collaborate on that single user’s request simultaneously.

In Data Parallelism (DP2), by contrast, GPU 0 handles User A while GPU 1 handles User B. At concurrency 1, User A only uses one GPU, with decode speed physically bounded by that single chip’s memory bandwidth (3.35 TB/s on H100 SXM).

Evaluating Cross-Node Tensor Parallelism

Seeing Together’s single-stream TP2 speed, we tested an experiment: what happens if you run TP2 across two separate 1× H100 cloud instances connected via standard datacenter virtual networking (VPC) rather than intra-node NVLink?

The result demonstrated the severe cost of network barrier latency:

Concurrency	DP2 (2 Separate Nodes)	Cross-Node TP2 (Over Network)	Single-Node TP2 (NVLink)	Network Interconnect Penalty
c = 1	73.71 tok/s	24.98 tok/s	189.61 tok/s	2.95x slower than DP2
c = 4	274.64 tok/s	56.47 tok/s	612.35 tok/s	4.86x slower than DP2
c = 8	483.19 tok/s	75.75 tok/s	1,033.44 tok/s	6.38x slower than DP2

At concurrency 8, cross-node TP2 throughput collapsed from 483 tok/s down to 75.75 tok/s. The physical explanation is straightforward:

The same two H100s at concurrency 8: NVLink inside one server, two independent replicas, or tensor parallelism stretched over the cloud network.

t_{step} = t_{GEMV} + 2 \times \frac{Bytes _{activation}}{Bandwidth _{interconnect}} + t_{latency}

During single-token decoding, GEMV computations for a 27B model take only 5 to 10 microseconds. Within a single chassis over NVLink, exchanging activations takes ~2 microseconds over 900 GB/s channels. Across separate physical servers over standard cloud networking, that same transfer requires 800 to 2,000 microseconds. The GPUs spent over 95% of their execution time stalled at network barriers waiting for TCP packets.

The Golden Rule of Topology: Never run Tensor Parallelism across physical machines unless you have dedicated multi-rail InfiniBand. If your GPUs reside on separate nodes, always deploy independent replicas with data parallelism (DP).

4. The Hardware Leap: What Happens on NVIDIA B200?

While H100 remains the workhorse of enterprise inference, next-generation NVIDIA Blackwell (B200) accelerators are entering production. We benchmarked Fireworks AI running Qwen 3.8 27B on a dedicated 2× B200 deployment:

Concurrency	2× H100 BF16 (Fireworks)	2× B200 BF16 (Fireworks)	Observed Hardware Speedup
c = 1	97.09 tok/s	153.38 – 161.32 tok/s	1.58x – 1.66x
c = 4	347.12 tok/s	571.82 – 597.11 tok/s	1.65x – 1.72x
c = 8	622.18 tok/s	979.49 – 1,001.32 tok/s	1.57x – 1.61x

Moving from H100 to B200 delivered a clean 1.6x to 1.7x throughput increase with zero code changes. This speedup is directly explained by memory hardware specifications:

NVIDIA H100 SXM features 3.35 TB/s of HBM3 memory bandwidth.
NVIDIA B200 features 8.00 TB/s of ultra-dense HBM3e bandwidth (a 2.38x hardware leap).

Because memory-bound autoregressive decoding scales near-linearly with memory bandwidth, B200 allows a 2-GPU cluster to cross the 1,000 tok/s barrier even in unquantized 16-bit precision.

5. Case Study: Reasoning Token Budgets Across API Gateways

During our evaluation of Qwen3.8-27B-FP8 on Doubleword’s public API, we encountered an instructive systems interaction that illustrates how reasoning-capable models interface with standard throughput benchmarks.

We configured our AIPerf test harness with standard parameters to measure decode throughput at a fixed length:

{"max_completion_tokens": 128, "temperature": 0}

While standard non-reasoning requests complete in 1 to 2 seconds for 128 tokens, these initial requests ran for approximately 247 seconds. An inspection of the returned payload counters explained the extended duration:

The 128-token cap applied only to the visible answer. The model first generated 15,774 hidden reasoning tokens, so each request ran for about 247 seconds.

Metric	Configured Parameter	Returned API Telemetry
Reasoning Tokens (<think>)	Default reasoning; cap requested	15,774 tokens
Answer Tokens	128 target	130 tokens
Total Generated Tokens	128 completion cap	15,904 tokens (full reasoning trajectory)

Gateway Semantics and Parameter Interpretation

Tracing the HTTP response headers revealed routing through an upstream gateway adapter. In many recent reasoning model architectures, API proxies interpret max_tokens or max_completion_tokens as applying strictly to the final visible answer, leaving internal reasoning tokens (within <think>...</think> tags) unconstrained.

When no explicit reasoning limit is enforced, the model executes its full natural thought trajectory (~15,774 reasoning tokens) prior to producing the concise final response.

Standardizing the Benchmark: To evaluate pure decode throughput at a controlled output length, Doubleword documents explicitly setting reasoning_effort: "none":

{"max_completion_tokens": 128, "reasoning_effort": "none"}

With explicit non-thinking parameters configured, Doubleword’s public API completed all 540 measured requests cleanly within the target token bounds:

Concurrency 1: 65.86 tok/s (TTFT p50: 708 ms, ITL: 12.4 ms)
Concurrency 4: 291.29 tok/s (TTFT p50: 725 ms, ITL: 12.5 ms)
Concurrency 8: 534.71 tok/s (TTFT p50: 708 ms, ITL: 12.7 ms)

Benchmarking Note on Token Accounting: When measuring reasoning models, it is essential to audit both internal thought tokens and final visible tokens. Standardizing reasoning parameters is necessary to ensure fair, reproducible throughput comparisons across providers.

6. Speculative Decoding: Tuning MTP4 vs. MTP8

Multi-Token Prediction (MTP) is one of the most effective ways to accelerate autoregressive decoding without loading a separate draft model. Instead of predicting a single token per forward pass, lightweight auxiliary heads speculatively propose multiple future tokens, which the main model verifies in parallel.

How multi-token prediction works: cheap heads guess several tokens ahead, the main model checks all guesses in one pass, keeps the matching prefix and adds one token of its own.

Our initial baseline used MTP1 (predicting 1 draft token ahead), yielding 613.54 tok/s at concurrency 8. We then ran an ablation study comparing MTP4 (4 draft tokens) against MTP8 (8 draft tokens) on identical pairs of H100 GPUs with FP8 weights and attention KV:

Profile	Concurrency	Mean Throughput (tok/s)	TTFT p50 (ms)	ITL p50 (ms)	End-to-End p95 (ms)	Draft Token Acceptance
g factor MTP4	1	133.00 [132.15–133.54]	253.95 ms	5.22 ms	1,221.62 ms	59.36%
g factor MTP4	4	437.41 [434.23–440.52]	290.94 ms	6.41 ms	1,407.07 ms	59.36%
g factor MTP4	8	769.13 [761.22–774.06]	303.63 ms	6.93 ms	1,666.06 ms	59.36%
g factor MTP8	1	129.81 [129.36–130.14]	259.32 ms	5.13 ms	1,423.34 ms	39.02%
g factor MTP8	4	417.11 [415.15–420.33]	302.78 ms	6.60 ms	1,601.21 ms	39.02%
g factor MTP8	8	767.94 [744.30–792.23]	315.16 ms	6.77 ms	1,883.54 ms	39.02%

Why did eight draft tokens fail to beat four?

Speculative decoding is a fundamental trade-off between verification speed and draft accuracy:

With MTP4, the model accepted 59.36% of all draft tokens. Every step generated an average of 3.37 accepted tokens with minimal verification overhead.
With MTP8, predicting 8 tokens ahead into the future proved significantly harder: acceptance plunged to 39.02%. While the average step size increased slightly to 4.12 tokens, the engine wasted GPU cycles generating and verifying rejected branches.
As a result, MTP8 yielded 2% to 4% lower throughput and increased p95 end-to-end latency by 13% to 17%.

MTP4 vs MTP8 at concurrency 4: deeper drafts are accepted less often, so the longer step does not pay for the extra verification work.

Conclusion: More speculative depth is not automatically better. For Qwen 3.8 27B, MTP4 is the empirical sweet spot, achieving 769 tok/s on 2× H100 without latency penalties.

7. Concurrency Scaling: Throughput Has a Latency Bill

Concurrency 8 is a standard comparison point, but it does not represent the saturation capacity of two H100s. We tested the scaling behavior of g factor under heavy loads of 16, 32, and 64 concurrent streams:

Concurrency	MTP1 Throughput	MTP4 Tuned Throughput	MTP4 TTFT p95 (ms)	MTP4 End-to-End p95 (ms)	Interactive Serving SLA
c = 16	1,013.98 tok/s	1,252.34 tok/s	530.00 ms	2,065.24 ms	Optimal interactive SLA
c = 32	1,604.98 tok/s	1,904.97 tok/s	794.38 ms	2,780.32 ms	High-throughput sweet spot
c = 64	1,645.97 tok/s	2,611.88 tok/s	1,433.17 ms	4,111.71 ms	Queueing delay boundary

2,612 tok/s on two H100s looks incredible on a slide, but you have to check the latency bill:

At c = 32, the cluster operates in its optimal production window: generating 1,905 tok/s while keeping p95 Time-To-First-Token well under 800 ms.
At c = 64, throughput expands to 2,612 tok/s, but prefill queueing delays cause p95 TTFT to climb to 1.43 seconds.

In production inference, peak throughput numbers are meaningless without latency percentiles. Always size your concurrency targets to your application’s TTFT service level objectives.

8. Prefix Caching & Prompt Speculation: Give Them Something to Reuse

In short-prompt benchmarks (~500 input tokens), prefix caching typically reports near-zero cache hits because each request presents completely unique text.

To evaluate how caching and speculation perform in real agentic workflows (where prompts share long system prompts, tool definitions, and conversation histories), we tested a workload with a 2,820-token shared prefix and a unique prompt tail (total input: 2,870 tokens):

Why the shared prefix helps twice: the cached prefix skips most of prefill, and suffix speculation copies tool-call text that already appears in the prompt.

Optimization Strategy	c=1 tok/s	c=4 tok/s	c=8 tok/s	Prefix Cache Hit Ratio	Draft Token Acceptance
Baseline (Cache Disabled)	57.92	209.92	349.84	Disabled	—
Automatic Prefix Caching	60.70	230.88	418.49	80.47%	—
Prefix Cache + N-gram (k=4)	172.62	558.05	1,046.40	83.76%	90.18%
Prefix Cache + Suffix Speculation	180.36	655.77	1,140.26	83.76%	99.12%

The empirical takeaways for long-context and agentic workflows are substantial:

Prefix Caching Eliminates Prefill Latency: Enabling prefix caching achieved an 80.47% cache hit rate, reducing concurrency-8 TTFT p50 from 855 ms down to 323 ms and delivering an immediate +19.6% throughput boost.
Context-Aware Speculation (N-gram & Suffix): When repetitive or structured patterns exist in the prompt, n-gram and suffix speculation propose continuations directly from the cached context without requiring an auxiliary neural net. This accelerated decode throughput by 2.7x to 2.9x, pushing throughput to 1,140 tok/s with a 99% draft acceptance rate.

Concurrency-8 throughput on a 2,870-token prompt with a 2,820-token shared prefix, as each optimization is switched on.

9. Architectural Rules for Honest Benchmarks

When evaluating published inference benchmarks or sizing dedicated clusters, five architectural dimensions determine real-world performance:

What was the GPU interconnect? Is that 1,000 tok/s running on a single server with 900 GB/s NVLink (TP2), or across independent nodes (DP2)?
What precision was actually served? Was it BF16, calibrated FP8, or aggressive 4-bit quantization? Has domain accuracy been qualified on target tasks?
Where was the client running? Were requests sent across the public internet or through internal cluster networking?
How were reasoning tokens budgeted? Does the token limit include the entire completion, and were internal thought chains isolated during speed testing?
What is the latency trade-off? High batch throughput is easy to achieve at high concurrency, but does the Time-To-First-Token still meet your interactive user requirements?

The Bottom Line: High-throughput inference is not a black-box commodity. It is a systems discipline that balances memory bandwidth, network topology, speculative decoding depth, and cache management. When engineered properly, a dedicated 2-GPU cluster can deliver world-class latency, break past 1,000 tokens per second, and provide complete data sovereignty at predictable costs. That is what we set up in a private LLM deployment.

Applying this to your own models? We deploy and tune open models on teams' own GPUs, in their VPC or on-premises. Private LLM deployment.

Originally published at g-ftech.com.

The Limits of AI: Induction, Deduction, and Why Models Can't Jump

Aleksei Romanov — Wed, 30 Sep 2026 20:05:28 +0000

Every week brings another breathless proclamation that Artificial General Intelligence (AGI) is just a few trillion tokens, a bigger datacenter cluster, or another reinforcement learning run away. The prevailing industry dogma assumes that intelligence is a single scalar curve: if you scale compute hard enough, everything—from writing bash scripts to inventing new branches of physics—will fall out of the loss function.

Yet in 2026, a thought-provoking position paper from Google DeepMind titled LLMs Can’t Jump, authored by research scientist Tom Zahavy (a key contributor to DeepMind’s AlphaProof system), planted a sharp flag in the ground. Zahavy made a precise, mathematically and epistemologically grounded argument that cuts through the marketing noise: today’s foundation models have mastered induction and deduction, but they are structurally incapable of abduction.

This is not a temporary bug that a larger context window or a cleaner pre-training dataset will fix. It defines the fundamental ceiling of current architectures. An AI can spot statistical regularities across trillions of words (induction). An AI can chain thousands of logical derivation steps within an established rulebook like Lean 4 or Python (deduction). What an AI cannot do is invent a brand-new axiom out of thin air when the rulebook itself is broken.

Below, we dissect the classical triad of reasoning first formulated by philosopher Charles Sanders Peirce, examine why LLMs excel at induction and deduction, explore the psychology of the abductive “aha!” moment through historic breakthroughs (Einstein, Kepler, Semmelweis), and define the true computational upper bound of machine intelligence.

Start here

Understand the three classical modes of inference (induction, deduction, abduction), why next-token prediction and RL verifiers hit a ceiling at paradigm shifts, and why human intuition remains indispensable for inventing new scientific axioms.

Induction: Statistical pattern generalization: extracting probable rules and regularities from observed data.

Deduction: Logical consequence: deriving guaranteed theorems and conclusions from established premises and axioms.

Abduction: Hypothesis generation: the creative leap that invents a novel explanatory premise or axiom to explain anomalies.

Axiomatic system: A formal framework defined by initial unproven assumptions (axioms) and derivation rules.

Paradigm shift: Replacing the foundational axioms of a discipline rather than computing consequences within them.

1. The Triad of Inference: Peirce’s Epistemology

More than a century before the first neural network was conceived, American philosopher and logician Charles Sanders Peirce (1839–1914) demonstrated that all human reasoning collapses into three distinct, non-interchangeable modes of inference:

Inference Mode	Logical Structure	The Classical Beanbag Example (Peirce)	Cognitive Function
Deduction	Rule + Case ⇒ Result	Rule: All beans in Bag A are white. Case: These beans are from Bag A. ⇒ Result: These beans are white.	Certainty: Computes the necessary consequences of established rules. Discovers no new truths.
Induction	Case + Result ⇒ Rule	Case: These beans are from Bag A. Result: These beans are white. ⇒ Rule: Probably, all beans in Bag A are white.	Generalization: Compresses statistical patterns from finite samples into empirical laws.
Abduction	Result + Rule ⇒ Case (Hypothesis)	Result: These beans on the table are white. Rule: All beans in Bag A are white. ⇒ Hypothesis: Perhaps these beans fell from Bag A.	Invention: Proposes a plausible explanatory cause for an unexpected, anomalous observation.

Notice the profound asymmetry between these three modes:

Deduction is truth-preserving: If your premises are sound and your logic is valid, your conclusion is guaranteed to be true. It is the language of mathematics, formal compilers, and logic verifiers. However, deduction cannot generate new premises; it can only unpack what was already latent in the starting axioms.
Induction is probability-increasing: As you observe more white swans, your confidence that all swans are white rises. It is the language of statistics, empiricism, and modern machine learning. However, induction is forever trapped in the distribution of what has already been observed.
Abduction is the only mode that introduces new ideas: It is the creative leap that posits an unobservable mechanism, a hidden variable, or an entirely new conceptual framework to explain why reality is behaving unexpectedly.

Peirce's three modes on the bean-bag example: deduction derives the result, induction generalizes the rule, abduction proposes the hypothesis that would explain what was seen.

2. Why LLMs Mastered Induction and Deduction

To appreciate why current AI hits a wall at abduction, we must first understand why it has been so spectacularly successful at the first two modes:

Induction: The Trillion-Parameter Pattern Compressor

Pre-training on the public web using the autoregressive cross-entropy loss is the most colossal inductive exercise in the history of science:

L_{pretrain} (θ) = - t = 1 \sum T lo g P_{θ} (x_{t} ∣ x_{1}, \dots, x_{t - 1})

By minimizing this loss over 15 trillion tokens, the model compresses the statistical regularities of human syntax, code patterns, common sense, and semantic associations into its weight matrices. When you give an LLM three few-shot examples and ask it to complete the fourth, you are watching pure statistical induction at work. It has mapped the empirical distribution of language better than any human ever could.

Deduction: Chain-of-Thought and Verifiable Reasoning

Over the past two years, reinforcement learning with verifiable rewards (RLVR) and test-time search (DeepSeek-R1, OpenAI o1/o3, AlphaProof) unlocked deep deductive mastery.

When you place a reasoning model inside a formal axiomatic system—such as Python syntax, Lean 4 type theory, or the rules of chess—the model does not need to invent new rules. The rules (the axioms) are fixed. The model uses Monte Carlo tree search, beam search, and policy gradients to explore branching deductive chains:

Axioms A \cup Premises P ⊢_{Lean 4} Theorem T

In formal theorem proving, DeepMind’s AlphaProof achieved silver-medal level performance at the International Mathematical Olympiad by chaining thousands of deductive tactic steps in Lean. A model can search millions of deductive paths per second without fatigue. Within a closed system with verified rules, AI is superhuman.

3. What Abduction Truly Is: The Leap Beyond the Box

If induction finds patterns within existing data, and deduction computes consequences within existing axioms, what is abduction?

Abduction is the invention of a new conceptual ontology when the existing rulebook cannot explain reality.

It is often called the “Aha!” moment or insight reasoning. Crucially, abductive leaps possess three characteristics that make them mathematically hostile to standard gradient descent and next-token prediction:

Non-Intuitive or Counter-Intuitive Connections: Abduction does not follow the path of highest probability. It frequently connects two domains that appear completely unrelated or contradictory under the current paradigm.
Inventing New Axioms, Not Solving Within Them: It does not adjust the weights of existing variables; it defines an entirely new variable or throws out a foundational assumption that everyone assumed was sacred.
Born from Sparse or Zero Evidence: An abductive leap usually occurs before the validating data exists. It is not an empirical summary of thousands of data points; it is a creative hypothesis designed to explain a singular, nagging anomaly.

Induction fits rules to observed data and deduction derives consequences of fixed axioms, both inside the current paradigm. Abduction answers an anomaly with a new axiom, which starts a new box.

The Paradigm Wall: In the words of philosopher of science Thomas Kuhn, language models excel at “normal science”—solving puzzles within an accepted paradigm using deduction and induction. Abduction is the spark of “scientific revolutions”—shattering the paradigm and replacing its foundations.

4. Historic Breakthroughs: Connecting the Counter-Intuitive Dots

To see why statistical transformers cannot “jump,” look at how humanity’s greatest intellectual leaps actually occurred:

Kepler, Semmelweis and Einstein each proposed the new idea before the evidence or theory that confirmed it: Rudolphine Tables and the 1631 Mercury transit, Pasteur and Koch, the 1919 eclipse.

Case 1: Albert Einstein and the Falling Elevator (1907)

In 1907, Isaac Newton’s laws of gravitation had reigned supreme for more than two centuries. They predicted the orbits of planets, the tides of oceans, and the trajectory of cannonballs with staggering precision. There was no raging experimental crisis. The slight discrepancy in the perihelion precession of Mercury (a microscopic 43 arcseconds per century) was considered a minor observational nuisance to be absorbed by an undiscovered asteroid.

A pattern-matching language model in 1907 trained on the entire corpus of physics would have assigned a 99.999% probability to the proposition: “Gravity is an attractive force proportional to mass and inverse-square distance.”

Then Einstein had what he called “the happiest thought of my life” (glücklichste Gedanke meines Lebens): “If a person falls freely, he will not feel his own weight.”

Through an embodied, physical thought experiment (a Gedankenexperiment of an elevator falling in a vacuum), Einstein made a wildly counter-intuitive abductive leap: gravity is not an attractive force at all. It is the geometric curvature of four-dimensional spacetime caused by mass and energy.

Notice: the empirical data required to validate General Relativity did not exist when Einstein formulated it. Arthur Eddington did not photograph the bending of starlight around the sun during a solar eclipse until 1919—twelve years later. Einstein didn’t perform induction on existing data. He invented a new axiom that told scientists what data to look for.

Case 2: Johannes Kepler and the Heresy of the Ellipse (1609)

For two thousand years—from Aristotle and Ptolemy through Nicolaus Copernicus and Tycho Brahe—astronomy rested on an unshakeable aesthetic and theological axiom: celestial bodies must move in perfect circles at uniform speeds.

When Copernicus placed the Sun at the center of the solar system, he did not abandon circles; in fact, he had to introduce more complex mathematical epicycles (circles rolling upon circles) than Ptolemy just to make circular orbits fit the sky.

When Johannes Kepler inherited Tycho Brahe’s peerless observational dataset on the orbit of Mars, he spent years trying to fit the data into combinations of circular epicycles. He managed to get the model to agree with observations across almost the entire orbit—except for a tiny discrepancy of 8 arcminutes (less than one-third the diameter of the full moon).

An optimizer would have treated 8 arcminutes as sensor noise or added another small epicycle. Instead, Kepler made the jarring abductive leap: nature does not care about our aesthetic preference for circles. Planets move in ellipses, with the Sun sitting at one focal point.

To 17th-century astronomers, an ellipse was an ugly, asymmetric, defective circle. It was completely counter-intuitive. Yet by shattering the circular axiom, Kepler collapsed decades of complex epicyclic math into three simple, elegant laws.

Case 3: Ignaz Semmelweis and the Invisible Poison (1847)

In the 1840s at the Vienna General Hospital, young mothers were dying of puerperal (childbed) fever at an apocalyptic rate of 10% to 18% in the First Obstetric Clinic (run by doctors and medical students), compared to only 2% in the Second Clinic (run by midwives).

The medical establishment dogmatically blamed the discrepancy on “atmospheric-cosmic-telluric influences” or “miasma” (bad air). When doctors tried to explain why the midwives’ ward had lower mortality, they suggested that the midwife clinic was less crowded, or that mothers in the first clinic suffered psychological trauma from hearing the priest’s bell during last rites.

Then, Semmelweis’s close friend, pathologist Jakob Kolletschka, died of identical symptoms after accidentally being pricked by a student’s scalpel during a post-mortem autopsy.

Semmelweis connected two completely disparate dots: medical students were walking directly from the autopsy room (dissecting rotting cadavers) into the maternity ward to examine laboring mothers without washing their hands. He posited an unheard-of hypothesis: invisible “cadaveric particles” were being transferred from dead bodies into living patients.

He ordered doctors to wash their hands with chlorinated lime before examinations. Maternal mortality plummeted from 18% to under 1.2% almost overnight. Yet because bacteria had not yet been formally identified under microscopes, the medical establishment mocked and ostracized him. Semmelweis had invented the germ theory of disease decades before Louis Pasteur and Robert Koch.

5. The Computational Ceiling: Why Scaling Won’t Produce Abduction

Why can’t we simply train a 10-trillion parameter model with reinforcement learning until it starts discovering new physics?

The answer lies in the mathematical constraints of the optimization loop itself:

1. The Verifier Paradox

In RLVR (reinforcement learning with verifiable rewards), a policy improves because an external verifier grades its outputs. But a verifier can only grade what its author already understands how to check.

If you set up an RL training gym for astronomy in the year 1600, your verifier code would calculate rewards based on how well trajectories aligned with Ptolemaic epicycles. When Kepler’s policy proposed an ellipse, the verifier would flag it as a violation of circular physics and assign it a reward of exactly 0! Reinforcement learning ruthlessly penalizes the very deviations that constitute scientific revolutions.

A verifier written in 1600 would reward one more epicycle and give Kepler's ellipse reward 0: RL can only reward what its verifier already knows how to check.

2. The Closed Symbolic Hull

An LLM operates within a discrete vocabulary embedding space (V ≈ 150,000). Every sentence it generates is an interpolation across the convex hull of human language. It is fundamentally a closed formal system.

As Tom Zahavy points out in LLMs Can’t Jump, human abductive leaps are deeply grounded in embodied physical sensation. Einstein didn’t manipulate symbols in an algebraic solver; he imagined the visceral sensation of falling in an elevator. Kekulé didn’t run an exhaustive combinatorial graph search over carbon bonds; he had a reverie of an ouroboros—a snake biting its own tail—and made the leap to the benzene ring.

Without physical grounding in a multi-modal world model governed by the invariant laws of matter and energy, pure statistical language models remain trapped in the “Chinese Room” of symbolic recombination.

6. The Division of Labor: What AI Does Best vs. What Humans Must Do

Recognizing the limits of AI is not pessimism—it is the hallmark of serious engineering.

We do not need language models to be Albert Einstein to unlock trillions of dollars in economic value. In modern civilization, 99.9% of real-world productivity is induction and deduction:

Synthesizing 50,000 lines of bug-free Verilog RTL from an architectural specification (Deduction).
Mining petabytes of genomics data for candidate oncology biomarkers (Induction).
Auditing multi-hundred-page commercial contracts against regulatory statutes (Deduction).
Compiling, testing, and optimizing high-throughput GPU kernels (Deduction & Empirical Induction).

These are the exact enterprise domains where platforms like g factor operate: giving organizations the power to post-train open models on their own proprietary workflows, equipping them with deterministic verification gyms (RLVR), and serving them with zero data leakage.

The division of labor for the next century of science and industry is crystal clear:

The Human Role: Identify the anomaly. Notice the 8-arcminute discrepancy that everyone else ignored. Wonder why doctors’ patients die while midwives’ patients live. Formulate the bold, counter-intuitive hypothesis. Make the abductive jump.
The AI Role: Take the new axiom and unpack its consequences across a million permutations. Verify the theorems, run the numerical simulations, test every corner case, and write the code. Execute the deductive and inductive marathon with superhuman speed.

The division of labour the article proposes: people notice the anomaly and propose the new axiom; models derive its consequences and test them at scale.

AI will not replace the spark of human genius. It will give that spark a multi-million-node engine to reshape reality.

Applying this to your own models? We build private RL training environments with deterministic verifiers for teams' own workflows. Private training environments.

Originally published at g-ftech.com.