DEV Community: Aleksei Romanov

Teaching a 27B Model to Write Trading Alphas: 101 Formulas, 12 Rewards and One Unseen Year

Aleksei Romanov — Sat, 10 Oct 2026 19:37:37 +0000

Qwen3.8-27B is trained with multi-reward RL to write one-line trading formulas in the language of WorldQuant’s 101 Formulaic Alphas. A deterministic verifier turns each formula into a daily dollar-neutral long-short book on 46 US stocks and scores it on 12 reward channels. On 50 prompts and a year of market data that training never saw, the mean score of its formulas went from −1.37 to +1.80, and an equal-weight book of them returned +16.2% with a market beta of 0.09.

This post documents the system: the formula language, how a formula becomes a portfolio, the difference between alpha and beta and how we keep the model from learning the market, each reward channel and the hack that created it, and the train and holdout results. The training recipe (GDPO, CISPO, REPO-R) and its experiment ladder are in Multi-Reward RL, Part 3, which described this gym with the domain hidden.

Open weights and data: the step-500 adapter from this post is on Hugging Face as Qwen3.8-27B-GFT-Quant-LoRA, and GFT-Quant-Alphas-SFT holds 512,544 verified brief-to-formula pairs from our RL runs for supervised fine-tuning.

Results of the best run (1790958861) on the 50 unseen prompts and the unseen year, before and after training.

Metric	Step 0 (untrained)	Step 500 (shipped)
Mean holdout score per formula	−1.37	+1.80
Valid formulas	45 / 50	49 / 50
Formulas with holdout score above zero	4 / 45	47 / 49
Equal-weight book: 1-year return after costs	−0.5%	+16.2%
Equal-weight book: Sharpe	−0.21	2.58
Book beta to the market	—	0.09 (correlation 0.19)

Start here

The formula language and where it comes from; the formula-to-portfolio pipeline; alpha vs. beta and the five layers that keep the model off the market's direction; the 12 reward channels, including a JEPA world-model stress test with a correlation-geometry objective; the reward hacks behind them; train vs. holdout results; checkpoint and book selection; limits.

Alpha: Return that comes from picking the right stocks relative to each other, independent of where the market goes.

Beta: Return that comes from exposure to the market's direction. β = 1 moves one-for-one with the market; β = 0 does not move with it.

Long-short book: Equal dollars long the top-ranked stocks and short the bottom-ranked ones.

CPCV: Combinatorial purged cross-validation: one formula scored on 15 different test slices of history; luck is penalised.

Turnover: The share of the book traded per day.

JEPA: Joint-Embedding Predictive Architecture: a model that predicts the representation of the future market state from the past, used here to generate stress scenarios.

0. System at a glance

Configuration of the run reported in this post.

Item	Detail
Policy	Qwen3.8-27B, NF4 weights + rank-32 LoRA, no visible reasoning
Action	One formula in <alpha>…</alpha>, at most 128 tokens
Verifier	Deterministic: parse → daily dollar-neutral book → costs → 12 reward channels
Universe	46 US large caps, daily OHLCV from June 2012
Train window	2012-06-29 → 2025-08-20 (3,304 sessions)
Holdout	2025-08-21 → 2026-08-21 (252 sessions), never used for training or selection
RL recipe	GDPO over 12 channels, CISPO loss, REPO-R, 12 answers per prompt, 600 steps, one H100
Checkpoint rule	Best train-window CPCV of the 50 probe answers (step 500)

1. The formula language

Source. In 2016 Zura Kakushadze published 101 trading alphas from WorldQuant, with the firm’s permission, as “explicit formulas – that are also computer code”. Eighty were in production at the time. They use daily open, high, low, close, volume, returns and vwap, have average holding periods of 0.6 to 6.4 days and an average pairwise correlation of 15.9%. The shortest:

Alpha#101: ((close - open) / ((high - low) + .001))

If a stock closed well above its open relative to its daily range, go long the next day: the paper calls it a delay-1 momentum alpha. Our DSL uses the same vocabulary, with a fixed contract:

The DSL contract. Every operator looks only backward in time.

Group	Element	Meaning
Fields	`open high low close volume`	daily prices and share volume
	`returns`	close-to-close return
	`vwap`	proxy: (high + low + close) / 3
Cross-sectional	`rank(x)`	rank of x across the 46 stocks on each day, in [0, 1]
	`scale(x)`	rescale so that sum(\|x\|) = 1
Time series	`delay(x, d)`	value d sessions ago
	`ts_delta(x, d)`	x[t] − x[t−d]
	`ts_mean, ts_std, ts_sum, ts_max, ts_min (x, d)`	rolling statistics over d sessions
	`ts_rank(x, d)`	share of the window below today's value
	`ts_corr, ts_cov (x, y, d)`	rolling correlation and covariance
	`ts_decay_linear, ts_ema, ts_skew, ts_kurt, ts_slope, …`	weighted averages and shape statistics
Element-wise	`+ − * /, abs, log, sign, power, min, max`	per stock, per day
Limits	`1 ≤ d ≤ 252`	total nested lookback ≤ 252 sessions
	`≤ 256 DSL tokens, depth ≤ 16`	the model's answer is also capped at 128 tokens
Safety	`recursive-descent parser`	nothing the model writes is executed as code

2. From formula to book

market data up to close(t)
  → evaluate formula          one value per stock
  → rank across stocks        values in [0, 1]
  → subtract the mean         weights sum to 0      (long $ = short $)
  → divide by sum(|w|)        gross exposure = 1    (0.5 long, 0.5 short)
  → fill at open(t+1)         earn open(t+1) → open(t+2), minus costs

The same three steps for six stocks with made-up values. Only the order of the formula's values matters.

Cost model. Holdout costs are deliberately harsher than training costs.

Stage	Costs
Training	3.5 bps per unit of turnover: 2 fee + 1 slippage + 0.5 half-spread, plus a small fixed impact
Holdout	Per-stock square-root market impact for a $1M book, capacity penalty, 50 bps a year to borrow the shorts

3. Alpha vs. beta: keeping the model off the market’s direction

Any daily return series can be split into a part that follows the market and a part that does not:

r_book(t) = α + β · r_market(t) + ε(t)

Beta is market exposure: a book with β = 1 gains 1% when the market gains 1%. Beta is cheap; an index fund sells it for a few basis points. Alpha is what remains when the market’s move is taken out: return from ranking stocks correctly against each other. A model that “learns to trade” by holding the market would look skilled in a rising year and fail in a falling one, so the gym is built to pay for alpha only.

With equal dollars long and short, a market-wide move cancels; only the spread between the two sides is earned.

Five layers between the model and the market’s direction.

Layer	Mechanism	What it blocks
1. Construction	Ranks are centred and scaled: long dollars always equal short dollars. Measured net exposure on the holdout: 0.00.	Direct market exposure
2. Factor reward (R4)	Daily P&L is regressed on the market, size, 12-1 momentum, 5-day reversal and low volatility; the Sharpe those factors explain is penalised.	Indirect exposure, e.g. long high-beta stocks, short low-beta ones
3. Prompt constraints	Briefs forbid known proxies, e.g. “do NOT lean on long-horizon price sums — they proxy market beta”.	Beta by design
4. Motif guard (R10)	Pure price-level or price-momentum formulas with no predictive power lose their positive reward.	Slow tilts that ride a trend
5. Holdout check	Beta and correlation of the shipped book against the market on the unseen year.	Anything the first four missed

Measured on the holdout year. The equal-weight market of the same 46 stocks rose +27.3%. The book returned +16.2% with β = 0.09 and a correlation of 0.19, so the market explains only about 2–3 points of its return. On the 7 days when the market fell more than 1.5% (on average −1.9%), the book lost on average 0.3%. In a strongly rising year a long-only index made more money; the book made less but independently of the market, with a higher Sharpe (2.58 vs. 1.95) and half the drawdown (−4.3% vs. −8.6%).

Left: growth of $1 for the book and for the market over the unseen year. Right: each dot is one day; the book's slope against the market is 0.09, close to flat.

4. The task

The task contract.

Item	Detail
Input	A design brief: mechanism, horizon, input rule, structural hint, forbidden pattern, risk preference
Output	One formula in `<alpha>…</alpha>`
Group	12 answers per brief, ranked against each other on every channel
Prompts	500 generated briefs → 226 kept: those the untrained model solved in some but not all of 8 tries

A real brief from the training set:

Market: US-listed stocks trading on weekday sessions. Design an alpha whose PRIMARY mechanism is: mixed-signal extremes (rolling max or min of a price-volume composite). Target horizon: roughly 168-day windows. Inputs: must use exactly one ts_corr. Structure: take the sign of a time-series relationship. Constraint: do NOT lean on long-horizon price sums — they proxy market beta. Risk preference: make it robust to a volatility-regime change.

Briefs the model always or never solves give no learning signal. Dropping them was worth +0.47 on holdout in Part 3.

5. Twelve reward channels

GDPO standardises each channel within the group of 12 answers, multiplies by its priority and sums, so no channel can dominate through its numeric range. Two hard rules follow: an invalid answer never receives a positive advantage, and each group stays zero-sum.

Channels grouped by the question they answer, with their GDPO priorities in the best run.

The verifier matrix. Formulas are the raw channel values before GDPO standardisation.

ID	Channel	Priority	Computes	Blocks
R1	`cpcv`	3.0	mean − 0.5·std of net Sharpe over 15 CPCV splits	luck; fit to one period
R2	`pbo`	1.0	penalty when a formula ranks far better in-sample than out of sample within its group (≥ −0.48)	overfitting
R3	`jepa_stress`	0.4	score drop in 4 JEPA-generated markets, log-compressed (≥ −1)	fragility to regime change
R4	`factor_neutrality`	0.4	−0.4 · clip(raw Sharpe − residual Sharpe, 0, 3) after 5 known factors	market beta and known factors in disguise
R5	`turnover`	1.0	−2 · max(0, turnover − 0.5)	books that pay their profit away in costs
R6	`min_turnover`	0.5	penalty below 6% a day; invalid below 2% a day	static and empty books
R7	`ic`	0.2	0.2 · clip(20 · IC, −1, 1); at most 0.05 while CPCV ≤ 0	profit without prediction
R8	`novelty_pnl`	0.4	0.4 · (1 − max \|corr\| of daily P&L vs. a 1,024-entry archive); duplicates penalised	clones of known books
R9	`novelty_syntax`	0.2	bonus for an unseen canonical form (windows bucketed) × (1 − P&L corr); −0.3 per duplicate	one template with jittered windows
R10	`motif_guard`	0.3	bare one-motif or price-only formula with IC < 0.005 loses its positive reward	lazy single-motif formulas
R11	`complexity`	0.2	−0.02 per expression node beyond 24	bloated formulas
R12	`format`	1.0	+0.05 for a reasoning block; always 0 in this run	(inactive without reasoning)

R1 · CPCV core score

The train window is cut into 6 blocks; every pair of blocks is one test slice (15 slices), with 10 days purged and 5 embargoed at the edges. For each slice the annualised net Sharpe is computed:

CPCV = mean(SR_1 … SR_15) − 0.5 · std(SR_1 … SR_15)

Example: Sharpe 0.8 on every slice scores 0.8. Sharpe 3.0 on three slices and −0.2 on twelve averages 0.44 but scores −0.20. Missing tag: −1.0. Unparseable formula or lookback over 252: −0.8.

Fifteen test slices from six blocks. The formula has nothing to fit, so each slice is an independent exam; the grey blocks are the in-sample part used by R2.

R5, R6 · Turnover band

Free between 6% and 50% of the book a day; penalised on both sides; invalid below 2%.

Example, a real answer of the untrained model:

scale(ts_rank(ts_delta(volume, 23), 47) - ts_rank(ts_delta(volume, 23), 47))

The expression subtracts a value from itself, so every weight is zero every day: no trades, no costs, no losses. Under high costs this beats any honest formula. Its turnover is 0, below the 2% floor, so it is invalid.

R3 · JEPA stress test

A JEPA world model is trained only on the train window (2012 to August 2025) and then frozen. It reads a window of the 46-stock panel with 25% of the stock-day cells masked in spans of 8 days, encodes one latent per stock, and predicts the latent of the next window. The target is an exponential-moving-average copy of its own encoder, so the model learns to predict market states rather than raw prices.

A second target comes from Observable Matrix Dynamics (OMD; Halperin, 2026), a dynamical-systems view of the market: it follows the trajectory of a fixed-size matrix built from rolling stock correlations and of its spectrum, and finds that the matrix’s effective dimension collapses in crises such as 2008 and 2020. We use the same idea as an auxiliary loss: the predicted latent must also recover the geometry of the next window’s correlation matrix C of the 46 stocks.

OMD-style geometry targets of the JEPA (loss weight 0.1). Computed on training data only.

Target	Computed as	What it captures
Mean correlation	average off-diagonal entry of C	how much stocks move together
Market-mode share	λ_max / trace(C)	weight of the single market factor
Effective factor fraction	(Σλᵢ)² / Σλᵢ² divided by 46	effective dimension of the market; collapses in crises
Residual factor fraction	the same without the market mode	diversity left beyond the market
Projector drift	rotation of the top-3 eigenvector subspace from the past to the next window	rotation between sectors and themes
Leading spectrum	shares of the top-5 eigenvalues	shape of the dominant modes

At reward time the latent is shifted along four channels — dispersion between stocks, trend flip, correlation and volatility regime — and decoded into complete price and volume panels. The formula is re-scored in each world with the same CPCV and costs, and the average drop becomes the penalty −min(1, 0.4 · log(1 + mean drop)). A fifth channel, a crash, was removed because it ranked formulas worse than chance (AUC 0.43).

Training (top) and use (bottom) of the stress test. The geometry target teaches the world model how the market's correlation structure moves, not only how prices move.

Other channels in one line each

R2 stability: first in-sample and tenth out of sample within the group is penalised; the penalty grows with the drop.
R4 factor neutrality: a formula whose Sharpe falls from 1.5 to 0.3 after removing the five factors pays −0.4 × 1.2 = −0.48.
R7 IC: the daily correlation of yesterday’s weights with today’s returns; an IC of 0.05 earns the full bonus.
R8, R9 novelty: ts_mean(close, 20) and ts_mean(close, 21) count as the same idea; new code that trades an old book earns nothing.
R10 motif guard, R11 complexity, R12 format: hygiene; R12 is inactive because this run answers without reasoning.

6. Reward hack log

The first version had a profit score and a format check. Each row below is a way the model found to score well without a real edge, and the change that closed it.

Reward and protocol changes, July – August 2026, from commit history and run reports.

Date	Observed	Change
Jul 4	All answers collapsed onto one formula	Syntax novelty (R9)
Jul 5	Different code, same book	P&L novelty with an archive seeded by Alpha101-style formulas (R8); smoke test: 2 → 5 distinct valid formulas
Jul 6	Static books: slow price-level tilts with IC 0.000 took the top rewards (0.55–0.63)	IC reward (R7); not enough alone (0.63 vs. 0.34 for momentum), so a minimum-turnover penalty the same day: 0.63 → 0.25
Jul 7	High-CPCV books trading 0.2–0.6% of the book a day still won	Hard floor: below 2% a day = invalid (R6)
Jul 7–8	Formulas tuned to the exact scored history	JEPA stress test (R3): static books −12 offline, honest formulas ≈ 0; next day log-compressed, as 24% of groups had ≥ 2 formulas at the cap
Jul 9	69–83% of answers were one momentum template with jittered windows; every arm failed a residual-Sharpe check	Window bucketing (5/21/63/126/252) before novelty; factor-neutrality penalty (R4)
Jul 10	63–83% of invalid answers still got positive advantage	Validity as a hard constraint on the advantage
Jul 23	Checkpoint choice could leak the holdout	Selection by train-window CPCV only; CPCV floor on the advantage
Jul 26	Stress crash scenario ranked formulas worse than chance (AUC 0.43 vs. 0.55–0.64); the turnover band charged top formulas −0.43	Crash scenario removed; band moved to 3–6% a day (top decile now −0.09)
Aug 9	Signal and fill on the same close: a subtle look-ahead	Fill at the next open, earn open-to-open returns

Every row was found by reading the top-scoring formulas. A rising reward curve was the first symptom of each hack, not evidence of learning.

7. Train vs. holdout

The best recipe from Part 3: CISPO + REPO-R, LoRA, 226 prompts, 600 steps. Part 3 shows the ladder of experiments that led to it, from DAPO at −0.47 to this recipe at +1.80.

Every checkpoint on the 50 unseen prompts (greedy). Train CPCV is on 2012 – Aug 2025 and selects the checkpoint; the other columns are on the unseen year.

Step	Train CPCV	Holdout score	Valid	Formula families	Book return	Book Sharpe
0 (untrained)	−0.98	−1.37	45 / 50	45	−0.5%	−0.21
100	−0.36	−0.47	47 / 50	47	+2.6%	0.95
200	−0.15	−0.13	49 / 50	49	+4.3%	1.36
300	+0.29	+0.85	48 / 50	41	+10.9%	2.14
400	+0.23	+0.51	49 / 50	45	+8.3%	2.00
500 (shipped)	+0.82	+1.80	49 / 50	31	+16.2%	2.58
600	+0.69	+1.58	48 / 50	34	+15.2%	2.56

Left: the training core score the model optimises. Right: the same 50 unseen prompts scored on the train window (grey, the selector) and on the unseen year (orange).

Step 500 has the highest train-window CPCV (0.82) and was shipped on that basis; it also has the highest holdout score. Holdout and train-window scores move together but not in lockstep: step 300 is better on holdout than step 400, and step 600 is slightly worse than step 500.

Every valid answer to the 50 unseen prompts, scored on the unseen year. Dashed lines: means.

Answers of the step-500 model to unseen prompts. Train CPCV and turnover on 2012 – Aug 2025; Sharpe and return on Aug 2025 – Aug 2026 after realistic costs.

Formula	Train CPCV	Turnover / day	Holdout Sharpe	Holdout return
`rank(ts_max(volume / vwap, 11) / ts_mean(close, 21))`	0.95	3.1%	3.13	+19.3%
`rank(ts_max(volume / vwap, 34) / ts_std(returns, 34))`	0.85	2.8%	2.97	+17.8%
`rank(ts_std(volume / (high - low), 34))`	0.82	2.3%	2.96	+18.3%
`rank(ts_std(volume, 8) / ts_std(close, 8))`	0.59	14.9%	2.70	+15.7%
`rank(ts_std(volume, 17) / ts_mean(volume, 17) + ts_std(volume, 17) / ts_mean(volume, 17))`	−0.47	23.2%	−1.09	−7.6%

Convergence. All 49 trained answers use volume, and most rank stocks by the variability of their trading volume relative to price level or price volatility. Distinct formula families fell from 45 to 31. We have not established why this signal worked over this year; it is one idea expressed many ways, so 49 formulas are far fewer than 49 independent bets.

One equal-weight book per checkpoint: all valid answers, realistic costs, compounded over the unseen year.

8. Selection: which checkpoint, which formulas

Checkpoint. Chosen by train-window CPCV of the 50 probe answers; the holdout is computed once, afterwards (rule in place since July 23).

Formulas for a book. Ranking by CPCV and taking the top 50 is the obvious choice and a poor one. On our formula archives that book had a median member CPCV of 0.71–0.75 but behaved like 1.8–2.6 independent bets (pairwise P&L correlation 0.50–0.63), with the worst drawdown of the selectors tested. A correlation-capped selector:

sort candidates by CPCV, descending
book = []
for f in candidates:
    if turnover(f) < 0.02: skip
    if max |corr(pnl(f), pnl(g))| >= 0.5 for any g in book: skip   # redundant bet
    book.append(f)

On two archives it reached a Sharpe of 2.24 and 2.00 against 1.85 and 1.39 for reward-ranked picks, with about 20 effective bets instead of 6–12 and half the drawdown, although its members’ median CPCV was only 0.38. The books in this post use neither selector: they average every valid answer with equal weight.

9. Limits

Our evaluation protocol marks a single-year result like this as no claim: evidence that the system learns, not evidence of a durable edge.

Survivorship. The 46 stocks are today’s large caps followed back to 2012; companies that shrank, merged or failed are absent.
One year, one seed. The holdout is 252 sessions from one market regime; the run is a single seed.
A gap in the probe. 15 of the 49 trained answers trade less than 2% of the book a day on the train window. Training would mark them invalid; the holdout probe scores CPCV only and does not apply the floor.
Neutrality. The book is dollar-neutral with a measured β of 0.09, but not sector-neutral, and our vwap is a proxy.

Why no model predicts the market with certainty

A market price is a projection of the real world: what companies sell and spend, what central banks and governments decide, wars, weather, new technologies, and the expectations of millions of people about all of it. Price and volume data are a shadow of that process. When the world changes in a way the past does not contain, the shadow changes too, and a formula learned from yesterday’s shadow cannot know it in advance. In principle, a computer that encoded every fundamental change in the world and every process driving it could forecast prices with high confidence. In practice the future is not in the data, and markets also react to the forecasts made about them. An alpha is therefore a small statistical edge that decays, and the only fair test is the one used here: out of sample, after costs, with the expectation that it will eventually stop working.

Takeaways

The verifier is most of the work. Most channels exist to close a specific way of looking profitable without an edge; each was found by reading top-scoring outputs.
Separate alpha from beta explicitly. Dollar-neutral construction, a factor reward and a measured holdout beta together kept the book’s market exposure at β = 0.09.
Diversity must be selected for. RL converged on one family; a correlation cap, not the best scores, makes a book of independent bets.

Data and references

The JSON export contains the reward definitions, the JEPA stress configuration, the train curve, every checkpoint’s holdout and book results, the daily book and market returns on the holdout, and all 100 formulas from the untrained and step-500 models with train and holdout metrics. Recipe and experiment ladder: Multi-Reward RL, Part 3. Reward-design principles: the verifier design playbook.

Kakushadze, Z. (2016), 101 Formulaic Alphas, Wilmott Magazine 84, 72–80. Halperin, I. (2026), Observable Matrix Dynamics of Stocks, arXiv:2607.19005. Run 1790958861: Qwen3.8-27B, NF4 with a rank-32 LoRA, CISPO + REPO-R, GDPO, 600 steps, one H100. Holdout score per formula: CPCV-style Sharpe on the holdout year after realistic costs, minus an excess-turnover penalty, clipped to ±5, averaged over valid greedy answers to 50 unseen prompts. Market: equal-weight open-to-open return of the same 46 stocks.

Applying this to your own models? We build private RL training environments with deterministic verifiers for teams' own workflows. Private training environments.

Originally published at g-ftech.com.

Multi-Reward RL, Part 3: GDPO + CISPO + REPO-R at 27B, and the Advantage Floor That Stopped Learning

Aleksei Romanov — Fri, 09 Oct 2026 17:51:18 +0000

Part 1 explained how PPO, GRPO, DAPO and GDPO turn several rewards into one learning signal. Part 2 benchmarked the trainers on a 14B model for 50 steps and found a trap: CISPO won the training curves and lost on held-out tasks. This part asks what happens when the same ideas get a bigger model and twelve times more steps.

We trained Qwen3.8-27B for 600 steps on a single-answer task scored by 12 reward channels, and climbed a controlled ladder: DAPO, then CISPO, then CISPO with REPO-R, then fewer but better prompts. Then we moved the winning recipe to a harsher environment and found a guard in our own advantage pipeline that silently threw away 96% of the learning signal.

The result in one line: same model, same rewards, same 600 steps. Held-out score went from −0.47 with DAPO to −0.02 with CISPO, to +1.33 with CISPO + REPO-R, and to +1.80 with a smaller, filtered prompt set. The starting model scores −1.37.

Open weights and data: the best run in this post (CISPO + REPO-R, 226 prompts, step 500) is on Hugging Face as Qwen3.8-27B-GFT-Quant-LoRA, and GFT-Quant-Alphas-SFT holds 512,544 verified brief-to-formula pairs from our RL runs for supervised fine-tuning.

One training step in our trainer. Stages 2–5 are published methods or a small guard of ours; none of them changes the verifier or the rewards.

Start here

See what GDPO, CISPO and REPO-R each change inside one training step, how much each moved the held-out score of a 27B model, why identical training curves can hide a 2× holdout gap, and how a sensible-looking advantage guard can stop learning completely when the environment gets harder.

Advantage (A): How much better an answer scored than the other answers to the same prompt. Positive A makes its tokens more likely; negative A makes them less likely.

GDPO: Normalizes each reward channel inside the prompt group before adding them with priorities, so a channel with a wide numeric range cannot drown the others.

CISPO: A policy loss that caps each token's importance weight at 1.2 instead of switching the token off when its probability has already moved.

REPO-R: Rescales each token's advantage by how rare the token is, with a strength ζ set by an entropy thermostat.

Advantage floor: A guard that forbids positive advantage for answers whose core score is at or below a threshold.

Holdout score: The score of a frozen checkpoint on prompts and data the trainer never saw. Higher is better; zero is break-even.

1. Where Part 2 left us

On a 14B model and 50 steps, CISPO produced the highest training reward and one of the worst held-out scores. Adding REPO-R recovered a little (+0.0175), not enough to beat the untrained thinking baseline. Two questions stayed open: does the picture change with a larger model and a longer run, and which part of a combined recipe actually does the work?

So this time we changed one thing per run and judged every run the same way: a frozen held-out probe, and a checkpoint chosen without looking at it.

2. The setup, abstracted

We keep the domain out of this post on purpose; the reward structure is what matters. The policy reads a short specification (one of 500 prompts) and writes one structured answer of at most 128 tokens. A deterministic verifier parses it, runs it on a frozen dataset and returns twelve reward channels:

Core score: a cross-validated estimate of answer quality. This is the objective; GDPO priority 3.0.
Stability check: penalizes answers whose in-sample and out-of-sample ranks disagree; priority 1.0.
Action-cost penalty: every change in the answer’s output costs a fee.
Nine guards and shapers: a secondary predictiveness score, complexity, format, syntax and behavior novelty, a neutrality gate, a minimum-activity guard, a pattern guard and a stress test.

Held fixed in every run: Qwen3.8-27B in NF4 with a rank-32 adapter (alpha 64, FP32 master weights; LoRA unless a row says DoRA), no thinking, 4 prompts per step with 12 answers each, temperature 1.0, learning rate 1e-5, KL β 0.08, GDPO, 600 optimizer steps, one H100, seed 42. The REPO-R rows also change two loss settings, described in section 5.

Holdout: every 100 steps, a frozen probe of 50 unseen prompts is scored on a later slice of data that training never touched. Each run “ships” the checkpoint with the best train-side core score on the same probe, so the holdout number is never used to pick it.

3. GDPO in one minute

With twelve channels on very different scales, plain GRPO adds them first and normalizes the sum, so the widest channel decides almost everything. GDPO standardizes each channel inside the prompt group first, then adds them with explicit priorities. Every run in this post uses GDPO, with priorities core 3.0, stability 1.0, pattern guard 0.3, complexity 0.2 and 1.0 for the rest; Part 1 covers the math and an interactive demo.

GDPO decides which answers count as better. The next two ideas decide how hard each token moves once we know.

4. CISPO: cap the weight, don’t drop the token

An answer was sampled by a slightly older copy of the policy. For each token, the ratio r = π_new / π_old says how much more likely the current policy makes it. PPO, GRPO and DAPO keep updates safe by clipping: once a token’s ratio moves past the band in the direction its advantage pushes it, its gradient becomes zero for that step. CISPO (from the MiniMax-M1 report) keeps every token in the gradient and only caps its importance weight; we use a cap of 1.2.

DAPO:  loss = −min(r·A, clip(r, 0.8, 1.28)·A)
CISPO: loss = −stopgrad(min(r, 1.2)) · A · log π(token)

Gradient weight of one token as its probability moves. The clipped objective switches the token off past the band; CISPO keeps it in the gradient and only caps its importance weight at 1.2.

Why this matters for short structured answers: a handful of “fork” tokens decide what the answer does. Those are exactly the tokens whose probability moves fastest when a good answer is reinforced, so a clipped loss switches them off first. CISPO keeps teaching them, just more quietly.

Measured effect: swapping DAPO for CISPO, with everything else equal, moved the held-out score from −0.47 to −0.02, a gain of +0.45. In Part 2, CISPO overfit at 14B; here, with checkpoint selection by train-side core score and a 27B model, it generalized better than DAPO.

5. REPO-R: an entropy thermostat

RL tends to collapse a policy onto a few answers it already likes. Entropy, a measure of how many alternatives the policy still considers, falls, and exploration stops. REPO-R counters this per token: in a better-than-average answer, rare tokens get more credit; in a worse-than-average answer, rare tokens get less blame.

A > 0: A′ = A · (1 − ζ · log p)
A < 0: A′ = A · (1 + ζ · log p)
where p is the token’s probability and log p < 0

REPO-R's advantage multiplier: rare tokens get more credit in good answers and less blame in bad ones. At the 0.05 cap a 1-in-1,000 token gets ×1.35 or ×0.65; at our median ζ the effect is under 1%.

The strength ζ is not a fixed knob. A controller records the average entropy of the first five updates as its target (the “w5” in our run names). When entropy falls below target, ζ doubles; when it rises above, ζ halves, and it never goes below zero. Here is the real log of our best run:

Real controller log of the best run (1790958861). ζ sleeps near zero, doubles up to 0.05 while entropy is below target, then halves back. Median ζ = 0.00078; above 0.01 on 27% of steps.

Measured effect: adding REPO-R to CISPO moved held-out from −0.02 to +1.33, a gain of +1.35, the largest single step in the study. A caveat we take seriously: the REPO-R configuration also switches on full-token loss (every token enters the loss, not only the 20% with the highest entropy) and two optimizer passes per rollout batch. With ζ this small most of the time, part of the gain may come from those two changes. The ablation that would separate them is still on our list.

6. Results at 27B

CISPO beat DAPO

One-line loss change, everything else fixed. Part 2’s 14B, 50-step benchmark had favored DAPO.

REPO-R bundle: biggest step

Includes full-token loss and two passes per batch, which we have not yet isolated.

Fewer, better prompts

Keeping only prompts where the starting model sometimes, but not always, succeeds gave the best run.

Same curves, 2× holdout gap

With REPO-R and DoRA on both, the training curves overlap. Only the holdout separates them.

Holdout score of the shipped checkpoint at each rung, chosen by train-side core score, not by holdout. One seed per configuration.

Step of the ladder	Configuration	Shipped step	Holdout	Change	Versus
Start: DAPO loss	DAPO · LoRA · 500 prompts	600	−0.47	—	starting model: −1.37
Swap the loss for CISPO	CISPO · LoRA · 500 prompts	600	−0.02	+0.45	DAPO · LoRA · 500 prompts
Add REPO-R (with full-token loss, 2 passes)	CISPO + REPO-R · LoRA · 500 prompts	600	+1.33	+1.35	CISPO · LoRA · 500 prompts
Train on 226 selected prompts instead of 500 (best holdout)	CISPO + REPO-R · LoRA · 226 prompts	500	+1.80	+0.47	CISPO + REPO-R · LoRA · 500 prompts
Swap LoRA for DoRA	CISPO + REPO-R · DoRA · 226 prompts	500	+1.33	−0.47	CISPO + REPO-R · LoRA · 226 prompts
Swap CISPO back to DAPO (keeps REPO-R, DoRA)	DAPO + REPO-R · DoRA · 226 prompts	500	+0.69	−0.64	CISPO + REPO-R · DoRA · 226 prompts

DoRA did not help once REPO-R was on: +1.33 against +1.80 for the same recipe with LoRA. Without REPO-R it gave DAPO a small lift (−0.29 against −0.47). The best run peaked at step 500 and sagged slightly by step 600, which is why we ship by the train-side selector rather than the last step.

Training curves are not a verdict

Same adapter, prompts and REPO-R; only the loss differs. The training curves overlap, the held-out scores do not: +1.33 for CISPO, +0.69 for DAPO.

This is Part 2’s lesson again, at a larger scale: a trainer can look identical on the training reward and still learn something quite different. Every decision in this post is made on a frozen holdout, and every run uses the same probe and the same selection rule.

Every training curve

The holdout chart is what we decide on; the training curves are what the trainer saw. In Environment A, the REPO-R runs stay close on training reward and core score while their held-out scores spread from +0.69 to +1.80. Every run’s per-step values are in the JSON export linked at the end.

7. The advantage floor trap

Next we moved the winning recipe to Environment B: different data, the same verifier, and per-action costs about 19× higher. Same 27B model, same GDPO, CISPO and REPO-R, and the same prompt filter, which this time found so few prompts with any successful answer that it kept the 100 closest to break-even. The stress-test channel is off there: its world model failed its own quality gates on that data. The runs started, the curves wiggled, and almost nothing happened.

The cause was a guard we had added to GDPO ourselves. It keeps invalid answers, and answers whose core score is at or below a floor (0), from receiving positive advantage, while preserving the rule that advantages inside a group sum to zero. In Environment A this is a sensible safety net: good answers score above zero. In Environment B, the fresh policy almost never does, and only about 1% of answers clear the floor. A group in which all twelve answers are below the floor gets twelve zeros.

The logs made the damage plain. With the floor, 4% of answers carried any learning signal; KL to the starting model stayed around 0.02. Removing the floor, and changing nothing else, gave 100%, a KL rising to about 0.3, and a training reward that climbed from about −10 to −2.5.

Environment B, first data slice, one change: the advantage floor. Ten-step means; the no-floor run was still training at 339 of 600 steps.

All seven Environment B runs, across four data slices, tell the same story: every floor-at-zero run stays flat, every no-floor run climbs.

All seven Environment B runs across four data slices: with the floor at 0, 3–4% of answers carried any signal and reward stayed flat; without it, nearly all did and reward climbed.

Turning the guard off did not turn it off

Our first fix was to disable the guard. The trainer then falls back to a second safety net that clamps any positive advantage to zero when the core score is at or below zero: the same trap with a different name. What worked was keeping the projection, which still forces invalid answers to non-positive advantage, and moving the floor far below the environment’s score range. A floor is an assumption about the reward distribution. When the environment changes, the assumption has to be checked again.

Even in Environment A the floor was not free: 10–41% of advantages were zero across runs, more in the runs that learned slowly. Whether a lower floor would also help there is one of our next experiments. The held-out results for Environment B will follow once those runs finish their probes.

8. What we would do again

Log the share of non-zero advantages from step 1. In a healthy run it sits well above half. At 4% the trainer is idling while the GPU bill keeps running.
Treat every guard threshold as a property of the environment. Re-derive floors and clamps when costs, data or reward ranges change, and check what the fallback path does when you disable one.
Change one thing per run and judge on a frozen holdout. Ship the checkpoint chosen by a train-side selector, and use the holdout only to verify it.
Prefer fewer, informative prompts. Prompts the starting model always or never solves give little signal; filtering them beat training on all 500.
Name the recipe by its parts. GDPO + CISPO + REPO-R is a combination of published methods plus one guard of ours, and it should be credited that way.

Limits: one seed per configuration; one held-out slice; REPO-R measured as a bundle with full-token loss and two passes; Environment B holdout pending. Treat the gaps as strong signals, not final effect sizes.

Evidence and method references

The JSON export holds every logged optimizer step (reward, core score, cost, KL, entropy), the share of non-zero advantages per step, the REPO-R controller log and the holdout score of every checkpoint, with run IDs. The Markdown companion has the tables.

Methods: DeepSeekMath / GRPO; DAPO; GDPO; MiniMax-M1 / CISPO; REPO-R.

Applying this to your own models? We post-train open models on teams' own tasks with SFT and GRPO, and hand back the weights. LLM post-training with g factor.

Originally published at g-ftech.com.

Pushing the Limits: Extreme Inference Speedup of Qwen 3.8 27B on NVIDIA B300 (100 to 10k+ tok/s)

Aleksei Romanov — Wed, 07 Oct 2026 17:00:16 +0000

Deploying a 27-billion parameter reasoning model like Qwen 3.8 27B on modern hardware presents a stark paradox. If you boot a default configuration on an NVIDIA B300 SXM6 GPU and send a solitary stream, you will measure roughly 104 tokens per second. The GPU sits largely cold, constrained by memory bus round-trips for every single token it emits.

Yet the same model on the same B300 GPUs reached 9,825 tokens per second on our custom stack with eight GPUs and 256 concurrent streams, and a dedicated Fireworks AI deployment on the same eight GPUs reached 16,931 tokens per second. These are aggregate numbers across many streams, not the speed of one stream.

Getting there is not a matter of flipping a single “turbo” flag. It requires systematically dismantling five consecutive systems bottlenecks: memory bandwidth latency, token serialization, redundant prompt prefill, repetitive structure generation, and multi-GPU interconnect overhead.

Over the past several days on our research cluster, we ran over 1,000 controlled benchmark trials across 2×, 4×, and 8× NVIDIA B300 GPUs, exploring tensor parallelism, data parallelism, multi-token speculation (MTP4 vs. MTP8), prompt prefix caching, and suffix reuse. Below is the verified engineering telemetry and the practical playbook for scaling Qwen 3.8 27B in production.

Start here

Explore how Qwen 3.8 27B scales from 104 tok/s to over 10,000 tok/s on NVIDIA B300. Discover why Tensor Parallelism rules single-stream latency while Data Parallelism dominates batch serving, why MTP4 beats MTP8 on reasoning workloads, how prefix caching plus suffix decoding delivers 99.7% speculation accuracy, and why doubling from 4 to 8 GPUs only yielded a 10% throughput bump.

Autoregressive Decode Bound: In single-stream generation, the entire 27B parameter weight matrix must be read from HBM for every individual token, bottlenecking throughput on memory bandwidth rather than compute.

Tensor Parallelism (TP): Sharding individual matrix multiplications across GPUs over high-speed NVLink to cut per-device memory-fetch latency for a single sequence.

Data Parallelism (DP): Running multiple independent model replicas on dedicated GPUs, serving concurrent requests in parallel with zero inter-GPU sync.

Multi-Token Prediction (MTP): Speculative decoding with a small prediction module trained with the model; it drafts k future tokens that the main model then verifies in one pass.

Prefix Caching: Reusing computed KV cache blocks for identical prompt segments across independent requests to bypass prefill compute.

Suffix Decoding: Speculating entire candidate phrases based on previously observed prompt or output n-gram histories.

1. The 104 tok/s Baseline: Why Raw Models Crawl

To understand how to make an LLM run at thousands of tokens per second, you must first understand why it runs at 100 tokens per second out of the box.

During the autoregressive decode phase, an LLM generates one token at a time. To emit a single token for a 27B parameter model in FP8 precision (~27 gigabytes of weights), the GPU must read those 27 GB from high-bandwidth memory (HBM) for every decode step. That sets a floor on the time per step; kernel launches, attention over the KV cache and scheduling add more. Without speculation, one stream measured 104 tok/s on a single-GPU replica and 135 tok/s with the model split across two GPUs.

time per decode step ≈ model weight bytes ÷ HBM bandwidth + kernel and scheduling overhead

At concurrency 1, most of the GPU’s compute sits idle: the execution units wait on memory loads. Therefore, scaling inference throughput requires solving two orthogonal engineering goals:

For Low Concurrency (Interactive Chat & Agent Tooling): How do we minimize latency per token so a human or autonomous agent experiences instant responses?
For High Concurrency (Batch APIs & Synthetic Data Generation): How do we pack requests together so that each memory-load pass over the weights computes tokens for dozens of requests simultaneously?

2. The Speedup Journey: From 100 to 16,900+ tok/s

We took a disciplined, step-by-step approach. Each experiment introduced a single architectural intervention: slicing layers across NVLink, deploying speculative draft heads, caching repeated prompt prefixes, mining repetitive output structures, and scaling independent data-parallel replicas.

The chart below shows the main steps of our tests on Qwen 3.8 27B across 2×, 4×, and 8× NVIDIA B300 GPUs:

The main steps, one change at a time (log scale). Going from 1 to 64 concurrent streams moved throughput the most. Our stack peaked at 9,825 tok/s; the 16,931 bar is Fireworks AI on the same eight GPUs.

Notice the distinct operational leaps along this curve:

The Latency Leap (104 → 305 tok/s): Tensor Parallelism (TP2 and TP4) combined with MTP4 speculative decoding cut inter-token latency by nearly 3×, pushing solitary stream speed to 305 tok/s.
The Concurrency Multiplier (305 → 4,464 tok/s): Bringing concurrency to 64 streams allowed continuous batching on 2 B300 GPUs to reach 4,464 tok/s with MTP4.
The Caching & Suffix Breakthrough (4,464 → 7,012 tok/s): On structured catalogs, prefix caching hit on 85% of prompt tokens, while suffix history predicted output sequences with 99.7% acceptance, breaching 7k tok/s on just two cards.
The Multi-GPU Expansion (7,012 → 9,825 tok/s): Scaling to 4 and 8 B300 GPUs with a balanced hybrid topology (DP4×TP2) pushed our custom stack to 9,825 tok/s.
A different stack on the same hardware (16,931 tok/s): A dedicated Fireworks AI deployment on the same 8 GPUs reached 16,931 tok/s at concurrency 256 on short prompts. It is a comparison point, not a further step of our stack.

3. Topology Dynamics: Tensor Parallelism vs. Data Parallelism

A frequent misconception in inference engineering is the dogma that “Data Parallelism is always best for throughput, and Tensor Parallelism is only for fitting big models in VRAM.”

With Qwen 3.8 27B FP8, the model easily fits onto a single B300 (288 GB of HBM). Yet when we benchmarked different allocations across 2, 4 and 8 GPUs, the reality proved far more nuanced. The table shows aggregate output tok/s on short prompts; the 2× rows are vanilla FP8 without speculation, the 4× and 8× rows use MTP4:

Hardware	Topology	Speculation	c = 1	c = 8	c = 64	c = 256
2× B300	DP2×TP1	Vanilla FP8	103.97	654.56	2,490.11	—
2× B300	DP1×TP2	Vanilla FP8	135.42	792.91	3,427.21	—
4× B300	DP1×TP4	MTP4	304.80	1,133.68	4,453.83	7,092.10
4× B300	DP2×TP2	MTP4	281.55	1,404.05	4,868.94	7,352.23
4× B300	DP4×TP1	MTP4	243.10	1,461.10	6,048.40	7,230.67
8× B300	DP1×TP8	MTP4	282.61	1,050.29	4,252.37	7,136.17
8× B300	DP2×TP4	MTP4	288.12	1,406.05	5,932.80	8,075.58
8× B300	DP4×TP2	MTP4	277.34	1,636.08	6,360.69	— (gateway timeout)
8× B300	DP8×TP1	MTP4	241.93	1,329.37	7,153.35	8,939.26

The Counter-Intuitive Vanilla Finding: On 2× B300 running Vanilla FP8 (without MTP), TP2 beat DP2 at every single concurrency level, including c=64 (3,427 tok/s for TP2 vs 2,490 tok/s for DP2). A likely reason: without speculation, sharding across NVLink shortened each forward pass enough for batches to cycle faster than on two slower independent workers. Once MTP4 was enabled, DP2 pulled ahead at c=64 (4,464 vs. 3,542 tok/s for TP2).

Key architectural principles from the topology sweeps:

Low Concurrency (c = 1 to 4): Tensor Parallelism is mandatory if user-facing latency matters. DP1×TP4 delivered 304.8 tok/s at c1 on 4× B300, compared to 243.1 tok/s for DP4×TP1. Every token decode step is shared over high-speed NVLink.
High Concurrency (c = 64 to 256): Data Parallelism dominates. At c=64 on 4 GPUs, DP4×TP1 delivered 6,048 tok/s, beating DP1×TP4 (4,454 tok/s) by +35.8%. Under heavy concurrency, TP’s AllReduce synchronization across GPUs on every layer is the likely cost.
The 8-GPU Limit (TP8): Never run TP8 across a full 8-GPU node for throughput. DP1×TP8 at c=64 delivered only 4,252 tok/s—slower than a 4-GPU system. The cost of 8-way AllReduce likely outweighed the extra compute.
The Long-Context Sweet Spot (DP4×TP2): On long prompts (2.8k tokens), the balanced hybrid topology—four replicas of TP2 pairs (DP4×TP2)—gave the best complete result at c=256, 9,825 tok/s, and also led eight single-GPU replicas (DP8×TP1) at c=1, 8 and 64. DP8×TP1 posted 11.2k and 11.6k tok/s in two of three repeats at c=256, but the third hit gateway timeouts, so we report no mean for it.

4× B300 with MTP4 on short prompts: one replica split over four GPUs gave the fastest single stream, four independent replicas gave the most total throughput at 64 streams.

4. Speculative Decoding: MTP4, MTP8, and the Hybrid Myth

Qwen 3.8 natively incorporates Multi-Token Prediction (MTP). Unlike traditional speculative decoding, which requires loading a separate smaller draft model (and paying memory bandwidth to read two sets of weights), MTP uses a lightweight prediction module trained together with the main model.

The MTP module drafts k future tokens, one draft step at a time, and the next forward pass of the main model checks all k at once. If each draft token is accepted with probability α, the expected number of tokens per verification pass is:

expected tokens per verification pass = (1 − α^(k+1)) ÷ (1 − α)

This is the standard estimate from Leviathan et al., 2023; the real speedup is lower because drafting and verification cost time.

We tested MTP with 4 draft tokens (MTP4), 8 draft tokens (MTP8), and a “Hybrid” engine that first attempts suffix matching before falling back to MTP. All rows ran on 2× B300 with DP2. Draft acceptance is accepted ÷ proposed draft tokens over the whole run; the c = 256 values come from a separate high-load run:

Speculation Mode	Workload Type	Draft Acceptance	c = 1 tok/s	c = 8 tok/s	c = 64 tok/s	c = 256 tok/s
MTP4 (Optimized)	Short Reasoning	59.09%	242.37	1,287.05	4,362.11	5,811.65
MTP8 (Optimized)	Short Reasoning	38.42%	240.87	1,272.54	4,122.65	—
Hybrid (Suffix → MTP4)	Short Reasoning	70.85%	199.58	1,064.13	3,784.48	—
Prefix + Suffix4	Long Structured	99.69%	325.53	1,936.26	5,704.54	7,011.98
MTP4 (Optimized)	Long Structured	99.99%	288.22	1,317.17	2,165.26	—
MTP8 (Optimized)	Long Structured	99.89%	368.55	1,520.00	2,227.49	—

The Lessons of Speculative Tuning

Why MTP4 beats MTP8 on reasoning: Reasoning workloads have high branch entropy. The model often changes course, backtracking or exploring logic. Under these dynamics, predicting 4 tokens ahead hit a healthy 59.1% acceptance rate. Pushing to 8 tokens (MTP8) caused acceptance to collapse to 38.4%. The compute spent generating and verifying rejected draft tokens 5 through 8 degraded throughput by over 200 tok/s at c=64.
When MTP8 wins: When sequences are highly structured or predictable (or when deep queue batching at c=256 keeps all workers saturated), MTP8 pulled ahead on 4 GPUs (8,239 tok/s vs 7,231 tok/s for MTP4, both DP4×TP1 at c=256).
The Hybrid result is mixed: In architecture discussions, engineers often propose: “Let us try suffix history first, and if that misses, fall back to MTP.” We built and measured this runtime. On TP2 it beat plain MTP4 at low concurrency (+7% on short and +12% on long prompts at c=1), but it lost in every c=64 cell, where throughput matters most. For batch serving, a single speculative method was the better choice.

5. Caching Strategies: Prefix Caching vs. Suffix Decoding vs. N-gram

In real-world applications, models rarely process isolated prompts. Enterprise workloads fall into two broad categories:

Ad-hoc High-Variance Prompts: Unique user queries, diverse reasoning tasks, or creative writing.
Structured Enterprise Prompts: System prompts, few-shot examples, JSON schemas, catalog extraction, and multi-turn agent tool loops.

We tested our dual-B300 stack with identical models under three caching regimes:

Workload	Cache Strategy	Prefix Hit %	Draft Accept %	c = 1	c = 8	c = 64
Short (564 in)	Cache Off	—	—	98.71	681.68	2,925.91
Short (564 in)	Prefix Cache Only	0.0%	—	95.71	656.24	2,871.04
Short (564 in)	Prefix + N-gram4	0.0%	78.98%	146.72	454.22	1,944.88
Short (564 in)	MTP4 Optimized	0.0%	59.09%	242.37	1,287.05	4,362.11
Long (2,870 in)	Cache Off	—	—	94.73	576.81	1,599.46
Long (2,870 in)	Prefix Cache Only	81.75%	—	95.98	659.48	2,851.20
Long (2,870 in)	Prefix + N-gram4	85.08%	90.17%	307.43	1,784.28	5,668.86
Long (2,870 in)	Prefix + Suffix4	85.08%	99.69%	325.53	1,936.26	5,704.54

Key Architectural Findings

Prefix Caching on Short Prompts is a Trap: If your prompts do not share a common prefix, enabling prefix caching provides a 0.0% hit rate while adding hashing and cache-lookup work. Throughput dropped from 2,925 to 2,871 tok/s at c=64. Do not turn prefix caching on blindly for randomized workloads.
The 1.78× Prefill Unlock: On long documents sharing a common system prompt or catalog base (2,820 shared tokens), prefix caching was transformative. Throughput at c=64 jumped from 1,599 to 2,851 tok/s (+78.3%), while p95 time to first token fell 63% (3,286 → 1,224 ms).
The Suffix 99.7% Acceptance Wonder: In repetitive workflows like catalog formatting or code generation, the model repeatedly outputs structured syntax. Suffix decoding stores and speculates matching n-grams from prompt and generation history. It achieved an astonishing 99.69% draft acceptance, catapulting long-context throughput to 5,705 tok/s at c=64 and 7,012 tok/s at c=256.

Prefix caching on 2× B300 at 64 streams: it never hit on short prompts with nothing shared, and added 78% on long prompts with a shared 2,820-token prefix. Suffix drafts on top more than tripled throughput.

6. Scaling 4 vs. 8 B300 GPUs: Where Efficiency Bends

One of the most consequential findings for platform teams is the scaling relationship between 4 and 8 B300 GPUs. When you double your hardware budget from 4× to 8× B300 SXM6, do you get a 2× throughput increase?

The hard telemetry answer is no. Under identical workloads, software stacks, and fixed concurrency, the scaling ratio was surprisingly modest:

Workload	Concurrency	Best 4-GPU Configuration	4-GPU tok/s	Best 8-GPU Configuration	8-GPU tok/s	Scaling Gain
Short Context	c = 1	MTP4 · DP1×TP4	304.80	Suffix4 · DP1×TP8	291.08	−4.5% (Regression)
Short Context	c = 8	MTP4 · DP4×TP1	1,461.10	MTP4 · DP4×TP2	1,636.08	+12.0%
Short Context	c = 64	MTP4 · DP4×TP1	6,048.40	MTP4 · DP8×TP1	7,153.35	+18.3%
Short Context	c = 256	MTP8 · DP4×TP1	8,238.99	MTP8 · DP8×TP1	9,207.40	+11.8%
Long Context	c = 1	MTP8 · DP2×TP2	452.68	MTP8 · DP2×TP4	436.26	−3.6% (Regression)
Long Context	c = 8	Suffix4 · DP4×TP1	2,193.59	MTP8 · DP4×TP2	2,427.24	+10.7%
Long Context	c = 64	Suffix4 · DP4×TP1	7,451.49	Suffix4 · DP4×TP2	9,153.70	+22.8%
Long Context	c = 256	Suffix4 · DP4×TP1	9,088.94	Suffix4 · DP4×TP2	9,825.28	+8.1%

The 8-GPU Cost Reality Check: Doubling your cluster from 4 to 8 B300 cards at c=256 yielded only an 8.1% to 11.8% throughput increase. 4× B300 already delivered 89% to 93% of the 8-GPU throughput. We did not test above 256 concurrent streams; up to that point, paying twice the hourly rate for an 8-GPU node bought at most 23% more throughput.

Best 8-GPU result vs. best 4-GPU result at the same workload and concurrency. Doubling the GPUs added at most 23%, and slightly slowed single streams.

Why did doubling GPUs not double throughput? We did not isolate the cause. Three candidates:

Gateway overhead: At 8,000+ tokens per second, the LiteLLM gateways and their Python event loops serialize JSON chunks and stream SSE events. Gateway CPU stayed near one core of four, which does not rule out a single-threaded bottleneck, and the 8-GPU deep-queue runs did hit gateway timeouts.
Client Serialization: Receiving and verifying 256 concurrent SSE streams under AIPerf could add client-side scheduling latency, although the 8-CPU client used only about 1.6 cores.
Queue Starvation at Fixed Concurrency: When 8 workers divide 256 concurrent requests, each worker averages only 32 active requests, which may not keep every B300 fully busy.

7. Head-to-Head: Fireworks AI vs. g factor Stack

We benchmarked our custom serving stack and a dedicated Fireworks AI deployment on the same GPU budgets, 2×, 4×, and 8× NVIDIA B300 SXM6, both serving Qwen 3.8 27B in FP8. Our stack quantized the BF16 checkpoint to FP8 at load time; we did not record which FP8 weights Fireworks used. Our stack was measured over in-cluster HTTP and Fireworks over HTTPS, so latency numbers include different network paths.

Neither system is uniformly faster across all workloads. Instead, the architectural trade-offs divide cleanly between interactive low-concurrency responsiveness and deep-queue batch saturation:

Hardware	Workload & System	c = 1 (Single Stream)	c = 8	c = 64	c = 256 (Deep Queue)
2× B300	g factor (MTP4 DP2 · Short)	244.91 tok/s	1,316.81 tok/s	4,464.12 tok/s	5,811.65 tok/s
2× B300	Fireworks AI (Dedicated · Short)	185.53 tok/s	1,138.03 tok/s	4,789.18 tok/s	6,953.51 tok/s
2× B300	g factor (Prefix+Suffix4 · Long)	325.53 tok/s	1,936.26 tok/s	5,704.54 tok/s	7,011.98 tok/s
2× B300	Fireworks AI (Dedicated · Long)	217.93 tok/s	1,345.03 tok/s	4,212.96 tok/s	5,637.99 tok/s
4× B300	g factor (MTP4 DP1×TP4 / DP4; MTP8 at c=256 · Short)	304.80 tok/s	1,461.10 tok/s	6,048.40 tok/s	8,238.99 tok/s
4× B300	Fireworks AI (Dedicated · Short)	186.74 tok/s	1,184.38 tok/s	5,745.23 tok/s	9,381.30 tok/s
4× B300	g factor (Prefix+Suffix4 DP4 · Long)	324.31 tok/s	2,193.59 tok/s	7,451.49 tok/s	9,088.94 tok/s
4× B300	Fireworks AI (Dedicated · Long)	218.27 tok/s	1,471.73 tok/s	6,549.06 tok/s	10,059.63 tok/s
8× B300	g factor (MTP4 DP2×TP4 / DP8; MTP8 at c=256 · Short)	288.12 tok/s	1,636.08 tok/s	7,153.35 tok/s	9,207.40 tok/s
8× B300	Fireworks AI (Dedicated · Short)	171.80 tok/s	1,300.37 tok/s	6,810.00 tok/s	16,930.96 tok/s
8× B300	g factor (DP4×TP2 Suffix4 · Long)	359.25 tok/s	2,361.05 tok/s	9,153.70 tok/s	9,825.28 tok/s
8× B300	Fireworks AI (Dedicated · Long)	218.47 tok/s	1,538.85 tok/s	8,499.90 tok/s	16,769.74 tok/s

Where g factor Outperforms

Single-Stream Interactive Decode (c = 1): g factor delivers 245 to 305 tok/s on short prompts, beating Fireworks (172 to 187 tok/s) by +32% to +68% at the same GPU count.
First-Token Response Latency (TTFT): At c=1, g factor’s p95 time to first token was 49–91 ms, compared to Fireworks at 237–940 ms. Part of that gap is the network: our stack was measured in-cluster, Fireworks over the internet.
Structured Long-Context Workloads (c = 1 to 64): On 2.8k-token contexts with shared prefixes, g factor’s Prefix Caching and Suffix4 speculation beat Fireworks by +8% to +64%, with the largest gaps at c=1 and c=8 (e.g., 2,361 vs 1,539 tok/s at c=8 on 8 GPUs).
Why: g factor leverages NVLink Tensor Parallelism (TP2/TP4) to accelerate individual token decodes, paired with MTP speculation and prefix caching. The Fireworks deployment ran one B300 per replica (no tensor parallelism) with 3 draft tokens.

Where Fireworks AI Outperforms

Massive Batch Saturation (c = 256): Fireworks surges to an incredible 16,931 tok/s on 8× B300, outperforming g factor (9,207 to 9,825 tok/s) by +71% to +84%.
4 → 8 GPU Scaling Linearity: Fireworks doubles its throughput smoothly as GPUs double (from 9,381 to 16,931 tok/s, a 1.80× ratio), whereas g factor scaled only +8% to +12% under our test harness.
Managed operations: Fireworks handled replicas and routing (with session affinity on) without any Kubernetes tuning on our side.
Why: We cannot see Fireworks’ internals. On our side, the 8-GPU deep-queue runs hit gateway timeouts, and gateway or client overhead is the leading suspect, but we did not isolate it.

Our stack vs. a dedicated Fireworks AI deployment on 8× B300. Ours was faster up to 64 streams; Fireworks pulled far ahead at 256.

8. The Systems Architecture Playbook: What to Deploy When

Based on our verified findings, here are four concrete architectural rules for deploying Qwen 3.8 27B in production:

Scenario A: Interactive Chat & Coding Agents

Workload: Concurrency 1–4, low latency critical (Time-To-First-Token and Inter-Token Latency).

Topology: TP2 (on 2 GPUs) or TP4 (on 4 GPUs).
Speculation: MTP4 enabled.
Prefix Cache: Off (unless long system prompts are shared).
Outcome: 280–305 tok/s single-stream speed, about 2.4–2.8 ms per output token.

Scenario B: High-Throughput Batch APIs & Synthetic Data

Workload: Concurrency 64–256, maximizing aggregate completion tokens per dollar.

Topology: Pure Data Parallelism (DP4×TP1 or DP8×TP1).
Speculation: MTP4 or MTP8.
Prefix Cache: Off for randomized data; on if prompts share templates.
Outcome: 7,000–9,200 tok/s on 4–8 B300 GPUs.

Scenario C: Document Analysis & Catalog Extraction

Workload: 2k–8k token context, common system prompt, structured repetitive outputs.

Topology: Balanced DP4×TP2 on 8 GPUs, or DP4×TP1 on 4 GPUs.
Speculation: Suffix4 (Prefix + Suffix cache).
Prefix Cache: Mandatory (85%+ hit rate saves massive prefill).
Outcome: 9,000–9,800 tok/s with 99.7% draft acceptance.

Scenario D: Cluster Sizing & Hardware Economics

Decision: Choosing between renting a 4× B300 node vs. an 8× B300 node.

Guideline: Start with 4× B300. It delivered 89–93% of 8-GPU throughput at c=256.
When to buy 8 GPUs: Only after measuring your own load above 256 streams; we did not test beyond that.
ROI: 4× B300 costs half as much per hour and gave up 7–11% of throughput at c=256 in our tests.

Configuring These Knobs in g factor: The main options from this benchmark (data and tensor parallelism, MTP4 or suffix speculation, prefix caching) are exposed in our CLI, SDK and Cloud Deploy interface, and we use the same stack for a private LLM deployment in your own VPC: gft deploy cloud Qwen/Qwen3.8-27B --dp 4 --tp 1 --speculation mtp4 --prefix-cache

All custom stack benchmarks conducted on dedicated NVIDIA B300 SXM6 AC nodes in Nebius uk-south1, September 20–21, 2026. AIPerf 0.12.0; our stack over in-cluster HTTP, Fireworks over HTTPS. Each cell is the mean of 3 repeats on a warm engine, not 3 cold starts. Our stack served about 300,000 measured requests; the 8-GPU run had 12 request errors in 3 cells, which we report as failed instead of averaging.

Applying this to your own models? We deploy and tune open models on teams' own GPUs, in their VPC or on-premises. Private LLM deployment.

Originally published at g-ftech.com.

Multi-Reward RL, Part 2: Benchmarking GRPO, DAPO, and CISPO on Unseen Tasks

Aleksei Romanov — Tue, 06 Oct 2026 20:25:13 +0000

Follow-up: Part 3 scales the CISPO + REPO-R recipe to Qwen3.8-27B and 600 steps, with a one-change-per-run holdout ladder and an advantage-floor failure we found in a harsher environment.

Part 1 analyzed how PPO, GRPO, DAPO, and GDPO balance competing reward objectives in theory. In this follow-up empirical benchmark, we test the practical question engineering teams actually face: which trainer algorithm actually produces a model that solves brand-new, unseen problems?

We ran seven distinct RL algorithms on Qwen3-14B in our deterministic decentralized exchange (DEX) arbitrage gym across two reasoning regimes: No-think (direct execution) and Thinking-2048 (with chain-of-thought tokens enabled). The results revealed a severe cautionary tale for post-training teams.

The Generalization Trap: In thinking mode, CISPO achieved the highest training reward curve (0.8951 last-10 mean), yet its performance on frozen test tasks collapsed to 0.6068. Meanwhile, the raw Qwen3-14B base model—with thinking enabled and zero RL training—scored 0.6830. If you only look at your training curves, you will celebrate a run that is actively degrading your model.

Holdout native score on 10 unseen tasks for each trainer, without thinking (blue) and with 2,048 thinking tokens (orange). Without thinking every trainer beats the untrained model; with thinking only DAPO-refill does.

Start here

Understand why training reward curves often diverge from holdout generalization in multi-step reasoning, see how Qwen3-14B performs across GRPO, DAPO, GDPO, CISPO, ADAPO, and REPO-R, why an untrained thinking baseline is mandatory before declaring victory, and examine the compute costs of test-time reasoning.

Holdout Generalization: Scoring the model checkpoint on completely frozen, unseen task parameters without updating weights on them.

Surrogate Inversion: When a policy optimizer ruthlessly exploits the composite training reward formula but produces worse actual domain solutions.

DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization): Token-level loss averaging with asymmetric clipping bounds (0.20 / 0.28).

CISPO: Clipped Importance Sampling Policy Optimization that caps importance weights directly at 1.20 rather than clipping surrogate ratios.

Thinking Baseline: Evaluating the raw base model with chain-of-thought tokens enabled, ensuring RL training adds genuine value beyond test-time compute.

1. What the agent does: The DEX gym

DEX stands for decentralized exchange. In our gym, the agent is presented with a frozen snapshot of automated market maker (AMM) liquidity pools: token reserves, exchange fees, gas prices, and routing constraints. The agent’s task is to find an optimal two-to-four pool arbitrage cycle and compute the exact input amount (amount_in_wei) that returns more of the starting asset after all fees and gas deductions.

WETH (Wrapped Ether) is the asset at both ends of the route. Consider an intuitive triangular arbitrage cycle: the model starts with WETH, swaps into USDC, swaps USDC into UNI, and swaps UNI back into WETH.

If the trade size is too small: The fixed Ethereum gas fees eat 100% of the price discrepancy, yielding a negative net return.
If the trade size is too large: The constant-product liquidity formula (x · y = k) causes severe price slippage, collapsing the margin.
If the pool sequence is invalid: The simulated EVM contract reverts and the transaction fails immediately.

The agent interacts using structured tool calls: inspecting pool reserves, proposing a pool route and trade amount, evaluating feedback, and finalizing. The verifier replays the transaction bytecode using a native Foundry / revm EVM execution engine. It scores the verified mathematical execution on the simulated blockchain, rather than trusting any self-reported text output by the model.

Training reward = R_native + 0.05 × R_efficiency + 0.20 × R_reliability

The composite training reward consists of three distinct channels:

Native Task Score (R_native ∈ [0, 1]): The verified net profit margin returned to the starting wallet after all pool fees and gas costs.
Execution Efficiency (R_efficiency ∈ [0, 1]): Rewards solving the task compactly in 2–3 tool turns instead of exhausting all 6 turns: max(0, 1 − tool_steps / 6).
Reliability Gate (R_reliability ∈ {-1, +1}): +1.0 if the transaction executed cleanly without syntax errors or contract reverts; −1.0 if the call failed.

The training reward adds efficiency and reliability bonuses to the native task score, so it can move without the task score moving. Holdout results report the native score alone.

2. What we held fixed: The training contract

To isolate the impact of the RL trainer algorithms, every other variable was strictly controlled:

Model and Quantization: Qwen3-14B (revision 40c06982…), quantized NF4 weights with BF16 compute, LoRA rank 32, alpha 64.
Compute Hardware: One dedicated NVIDIA RTX PRO 6000 Blackwell 96 GB per training run, Hugging Face generation, fused SDPA attention, CUDA graphs disabled.
Optimizer Schedule: Fixed seed 42, learning rate 1e-5 with linear decay, 50 optimizer steps, group size 4, eight completions per fresh rollout batch, two policy updates per rollout. Curriculum learning disabled.
No-think mode: Chain-of-thought thinking disabled; max 1,024 generated tokens per tool turn (max 8,192 trajectory cap).
Thinking-2048 mode: Up to 2,048 thinking tokens inside a 3,072-token turn budget (max 22,528 trajectory cap).
Frozen Holdout Evaluation: Evaluated on the final step-50 checkpoint. 5 task families × 2 held-out seeds × 4 samples = 40 episodes across 10 distinct unseen scenarios. Untrained starting checkpoints evaluated identically in both reasoning modes.

3. Trainer features: Seven algorithm bundles

In modern post-training, an RL algorithm is not a monolithic block. It is a bundle of three distinct modular decisions: which rollouts enter the training batch, how multiple rewards are combined into policy advantages, and how that advantage scales policy gradient updates.

Configuration	Loss / averaging	Reward → advantage	Dynamic sampling	Control
GRPO	Clipped surrogate; sequence-level averaging	Joint reward → group normalization	Off	Monitor only
DAPO	Clipped surrogate; token-level averaging	Joint reward → group normalization	Off	Fixed asymmetric clip: 0.20 / 0.28
DAPO-refill	Same DAPO loss	Same as DAPO	On; at most 2 refill rounds	Same fixed clip
GDPO	DAPO loss in this implementation	Normalize each reward, then combine	Off	Fixed asymmetric clip
CISPO	Detached, capped importance weights	Joint reward → group normalization	Off	Importance-weight cap 1.20
DAPO + ADAPO	DAPO loss	Same as DAPO	Off	Entropy feedback adjusts upper clip
DAPO + REPO-R	DAPO loss	Token-level advantage shaping	Off	Entropy feedback controls ζ
CISPO + REPO-R w5	CISPO loss; cap stays 1.20	Token-level advantage shaping	Off / separate refill arm	5-step target window; ζ ≥ 0

GDPO decouples reward-channel normalization before combining advantages. CISPO replaces ratio clipping with direct importance weight clipping (capped at 1.20). ADAPO and REPO-R introduce adaptive entropy control: ADAPO adjusts clipping bounds based on policy entropy, while REPO-R shapes token-level advantages. DAPO-refill activates replacement sampling when a rollout group exhibits zero reward variance, preventing zero-gradient wasted batches.

4. Full results: Train vs. holdout test

The CISPO Inversion Trap

CISPO dominated training reward curves but collapsed on held-out tasks. One likely reading: it kept raising the composite training reward without improving the native score on tasks it had not seen.

Thinking Baseline Beats RL

Untrained Qwen3-14B with 2,048 thinking tokens beat 6 of the 7 trained models. Always benchmark against an untrained thinking baseline.

DAPO Leads Generalization

DAPO delivered the highest native holdout score in no-think mode (0.6467, 39/40), and DAPO-refill led thinking mode (0.7105, 38/40).

The 5× Compute Tax

Thinking steps took 4.5–5.9× longer wall-clock time. You must balance the evaluation gain (+0.0275) against a 5× compute budget increase.

Thinking mode: training reward over the last ten points against holdout score. CISPO had the highest training reward and the lowest holdout score.

No-think mode: Every trained model beats the starting baseline

Configuration	Steps	Train mean	Train last-10	Holdout native	Successes	Step, s	Step-hours
Starting model (baseline)	0	—	—	0.3856	16/40	—	—
GRPO	50/50	0.7038	0.7729	0.6424	38/40	177	2.46
DAPO (top holdout)	50/50	0.6519	0.7142	0.6467	39/40	176	2.45
DAPO-refill	50/50	0.6564	0.7585	0.5839	34/40	224	3.11
GDPO	50/50	0.6488	0.7727	0.6245	36/40	167	2.32
CISPO	50/50	0.6486	0.7486	0.6190	36/40	164	2.27
DAPO + ADAPO	50/50	0.6694	0.7525	0.6332	37/40	168	2.33
DAPO + REPO-R	50/50	0.6373	0.7057	0.5724	31/40	156	2.16

Without reasoning tokens, post-training delivers an unambiguous leap forward. The untrained base model scored only 0.3856 (passing 16/40 episodes). DAPO achieved 0.6467 (39/40 successes), closely followed by GRPO at 0.6424. Every single trained configuration substantially outperformed the starting model.

Thinking-2048 mode: The starting model sets a high bar

Configuration	Steps	Train mean	Train last-10	Holdout native	Successes	Step, s	Step-hours
Starting model (baseline)	0	—	—	0.6830	37/40	—	—
GRPO	50/50	0.8362	0.8824	0.6798	38/40	929	12.90
DAPO	50/50	0.8036	0.7916	0.6703	38/40	938	13.02
DAPO-refill (top holdout)	50/50	0.8275	0.8564	0.7105	38/40	1016	14.10
GDPO	50/50	0.8190	0.8253	0.6602	35/40	938	13.03
CISPO (diverged)	50/50	0.8388	0.8951	0.6068	33/40	907	12.60
DAPO + ADAPO	50/50	0.8185	0.8815	0.6250	34/40	887	12.32
DAPO + REPO-R	50/50	0.8285	0.8206	0.6461	34/40	922	12.81
CISPO + REPO-R w5	50/50	0.8165	0.8332	0.6243	34/40	937	13.01
CISPO + REPO-R w5 + refill (partial)	41/50	0.7908	0.8302	—	—	912	10.39

When chain-of-thought thinking tokens are enabled, the starting model achieves 0.6830 straight out of the box (passing 37/40 episodes). Only DAPO-refill surpassed the untrained model on native holdout reward (0.7105, an incremental gain of +0.0275). GRPO (0.6798) and standard DAPO (0.6703) finished slightly below the base model’s score despite high success rates (38/40), demonstrating that success counts and solution quality answer different questions.

5. Explore the curves: Rollout telemetry

Each 50-step run generated 25 fresh rollout reward checkpoints (even optimizer steps update on rollouts generated on odd steps). The chart below shows three of them in thinking mode:

Raw training reward in thinking mode for CISPO, DAPO and DAPO-refill. One batch of eight completions per point, so the curves are noisy; the late averages and the holdout scores tell different stories.

Over its last ten reward points CISPO averaged 0.895 in thinking mode and DAPO 0.792, yet on unseen tasks DAPO scored 0.670 and CISPO 0.607. The training reward is measured on training tasks and includes bonuses, so it is no substitute for a frozen holdout.

6. Time and memory: The cost of thinking

Test-time reasoning comes with a heavy computational invoice. Across all seven algorithms, an average thinking step took 4.5–5.9× longer than a no-think step. Fifty training steps required approximately 12.3–14.1 hours of GPU execution in thinking mode, compared to 2.2–3.1 hours in no-think mode.

Mean seconds per optimizer step. Thinking made steps 4.5 to 5.9 times slower, for at most +0.028 over the untrained model on holdout.

Configuration	Peak allocated, GiB	Peak reserved, GiB
GRPO	69.1	76.9
DAPO	66.5	74.0
DAPO-refill	68.5	76.3
GDPO	68.3	76.1
CISPO	70.0	77.8
DAPO + ADAPO	71.8	79.6
DAPO + REPO-R	68.0	75.5
CISPO + REPO-R w5	66.9	74.4
CISPO + REPO-R w5 + refill (partial)	70.3	77.8

The table covers the thinking-mode runs: peak memory stayed between 66.5 and 71.8 GiB allocated (74.0–79.6 GiB reserved), within the 96 GB budget of the Blackwell GPU. The primary operational bottleneck is wall-clock rollout latency, not GPU memory.

7. Combining features: The CISPO + REPO-R hybrid

Can we rescue CISPO’s optimization speed while fixing its generalization collapse? We tested a hybrid architecture: using CISPO’s capped importance weights (cap 1.20) combined with REPO-R’s token-level advantage shaping.

To stabilize entropy dynamics, we introduced a nonnegative window-5 controller: the first 5 optimizer steps record entropy drift without intervention (ζ = 0). Once calibrated, the entropy control strength is constrained strictly to 0 ≤ ζ ≤ 0.05.

The completed hybrid achieved 0.6243 native holdout reward and 34/40 successes, outperforming standalone CISPO (0.6068, 33/40). While this +0.0175 lift confirms that advantage shaping mitigates policy collapse, the hybrid still trailed the raw untrained thinking baseline (0.6830).

Enabled is not the same as exercised

In no-think mode, DAPO-refill triggered 19 replacement rollout rounds. In thinking mode, DAPO-refill triggered zero refills—the model never produced an all-identical reward batch. Therefore, we cannot attribute the thinking DAPO-refill lead (0.7105) to the refill mechanism itself; it functioned identically to standard DAPO under that seed.

8. What to test next: Next iteration & replication

Based on these findings, our production post-training recommendations for multi-step reasoning workflows are:

In No-Think Mode: Standard DAPO and GRPO are clear, cost-effective winners. They raised the holdout score from 0.386 to about 0.64–0.65 (16/40 to 38–39/40 successes) in about 2.5 GPU-hours.
In Reasoning Mode: Always evaluate your untrained base model with reasoning enabled before declaring training success. DAPO-refill is the primary candidate for replication.
Never Pick a Checkpoint by Training Curves: CISPO proved that soaring training curves can mask test-set degradation. All promotion decisions must be gated by frozen holdout benchmarks.
Multi-Seed Replication: Expand seed sweeps (seeds 43, 44) across novel scenario families to confirm that DAPO-refill’s +0.0275 edge holds across market distributions.

Evidence and method references

All raw empirical telemetry is publicly available. The public JSON export contains complete run manifests, SHA256 hashes, all 791 recorded optimizer steps, 396 fresh reward logs, and 680 evaluated holdout episodes. The Markdown companion provides searchable per-step records.

Foundational algorithm publications: DeepSeekMath / GRPO; DAPO; GDPO; MiniMax-M1 / CISPO; ADAPO and REPO-R.

Applying this to your own models? We build private RL training environments with deterministic verifiers for teams' own workflows. Private training environments.

Originally published at g-ftech.com.

The Verifier Design Playbook: How to Build RLVR Gyms That Models Can't Game

Aleksei Romanov — Mon, 05 Oct 2026 22:25:40 +0000

In traditional machine learning, your loss function is a clean mathematical equation: mean squared error, cross-entropy, or cosine distance. The math is simple, deterministic, and impossible for the model to corrupt.

In Reinforcement Learning with Verifiable Rewards (RLVR), your verifier is your loss function.

The verifier is the Python script, Docker container, or compiler that runs after every rollout, inspects what the model just did, and hands back a numerical reward. If that verifier has a single logical loophole, the model will find it within two hundred training steps. It will exploit that loophole with relentless mathematical precision, report a 100% training reward, and produce a checkpoint that is completely useless in production.

Over the past year, engineering teams deploying RL post-training across coding, chip design, spreadsheets, and databases discovered that building the training harness is only 20% of the battle. The other 80% is verifier defense engineering.

Below is the pragmatic, battle-tested playbook for designing deterministic RLVR verifiers: how to structure the task contract, how to layer defense-in-depth, and how to protect your runs against the real-world exploits that derail production post-training.

Start here

Master the 3-layer verifier architecture (Gate, Oracle, Risk Scorer), prevent active inaction (the Zapier trap) and canned refusals, remove leftover caches and artifacts from rollout environments, apply asymmetric risk matrices, and run tamper-proof canary tests.

RLVR (Verifiable RL): Reinforcement learning where policy updates are driven strictly by programmatic outcomes (compilers, unit tests, linters, API response codes) rather than human opinion.

Fail-Closed Gate: A mandatory validation step that halts execution and awards exactly 0.0 reward if safety, protocol, or state-change invariants are breached.

Active Mutation Verification: Confirming that an agent actually caused verified state changes (HTTP 200 writes, DB row deltas) rather than observing a world that already met criteria.

Asymmetric Loss Matrix: Weighting failure modes unequally in the reward function (e.g. penalizing a missed compliance infraction several times more heavily than a false alarm).

1. First Principles: Code Over Opinions

The first rule of verifier design is deceptively simple: never use an LLM judge where a deterministic compiler or test script can run.

When you use another language model to score your agent (“Rate the quality of this answer from 1 to 5”), you invite catastrophic Goodhart’s law into your training loop:

Verbosity bias: The agent learns that generating 2,000 words of polite preamble tricks the evaluator into awarding higher scores.
Sycophancy: The agent learns to flatter the prompt’s assumptions rather than reporting uncomfortable factual errors.
Prompt injection: Under policy gradient pressure, models discover strings that confuse the judge into outputting a high score regardless of content.

A deterministic verifier has no taste and no feelings. It runs a test suite, checks a hash, compiles an abstract syntax tree, or verifies database state transitions. Did the query execute without syntax errors? Did the motor simulation stay within 50 N·m of torque? Yes or no. That cold, unambiguous verdict is the only foundation stable enough to support millions of policy gradient updates.

2. The Task Contract: Boundaries & Difficulty Calibration

A verifier cannot save a broken task definition. Before writing a line of scoring logic, you must pin down the Task Contract:

Pinned Seeds & Independent Oracles

Every training episode must generate its initial state from a reproducible seed using an independent “oracle” (an algorithmic script or verified human solution). Never generate dynamic training tasks using an unseeded random number generator or live external web scrapes: if your training distribution drifts during a 50-step run, optimization collapses into noise.

Semantic Splits vs. Row Shuffling

Never create your training and held-out evaluation sets by simply shuffling a CSV table. A model will easily memorize superficial column names or syntactic quirks. Split tasks by template families, independent seed ranges, and semantics-preserving transformations (e.g. permuting variable names, reordering table columns, or altering schema IDs while preserving the underlying logic).

Shuffling rows puts variants of the same template on both sides of the split. Holding out whole template families makes the held-out score measure skill instead of memory.

The Difficulty Sweet Spot: Why 0% and 100% Kill Training

Reinforcement learning algorithms like GRPO work by comparing multiple candidate attempts for the same problem. The attempts that score above average are reinforced; the attempts that score below average are discouraged.

The Math of RL Signal: If a task is too easy and the model solves it 8 out of 8 times, all advantages are zero. The model learns nothing. If a task is too hard and the model fails 8 out of 8 times, all advantages are zero. The model learns nothing. Useful policy gradients only occur in the struggle zone: where success is between 20% and 75%.

Before spending thousands of dollars on GPU clusters, run your base model across your task pool. Prune tasks that the model already solves 100% of the time (save your compute). For tasks where the model scores 0%, do not expect sparse RL to magically discover the answer—warm up the model with Supervised Fine-Tuning (SFT) demonstrations first.

Calibrate the pool first: tasks the base model never solves need SFT demonstrations, tasks it always solves can be pruned, and the 20–75% band carries most of the GRPO signal.

3. The 3-Layer Architecture: Defense-in-Depth

A production-grade verifier should never be a single monolithic script. It should be structured as three distinct, sequential layers:

Layer	Responsibility	Execution Rule	Failure Consequence
Layer 1: Hard Gate	Safety boundaries, format validation, active mutation proof, canary checks.	Binary (Pass / Fail), fast, fail-closed.	Instant reward = 0.0; halts execution immediately.
Layer 2: Native Oracle	Target compiler, physical simulator, database engine, or unit test suite.	Deterministic execution in volatile RAM (`/dev/shm`).	Computes raw correctness score (e.g. 8/10 unit tests passed).
Layer 3: Risk Scorer	Asymmetric risk weighting, milestone laddering, efficiency penalties.	Scales objective score based on business operational risk.	Encodes domain priorities (e.g. a heavier penalty on critical compliance misses).

This layered structure enforces a non-negotiable rule: Layer 3 can never rescue a failure in Layer 1. If an agent produces clean, elegant code that violates a security boundary or fails to cause an active state mutation, it receives exactly zero reward. No amount of eloquence or speed can compensate for a broken interface contract.

Three rollouts through a three-layer verifier. A failed gate ends the episode at 0.0; the oracle measures correctness; the risk scorer can lower a score for a critical miss but never rescue a failed gate.

4. Target-Native Engine Execution

When training agents on specialized technical domains—SQL dialects, BIM architecture, CAD geometry, or Verilog RTL—many teams make the rookie mistake of writing a Python regular expression or a lightweight mock simulator to grade answers.

This is a catastrophic trap. Language models are master regex-hackers: they will find syntax combinations that satisfy your regex while completely crashing when executed on a real database engine.

Always bind the verifier directly to the authoritative, target-native compiler or engine:

Enterprise SQL: Execute the generated query against a real, ephemeral database instance (e.g. Snowflake or Postgres). Check that the query executes without warnings and that the returned result table matches the gold dataset.
Hardware Design: Run actual EDA linters and simulators (Icarus, Verilator, Yosys). Formal equivalence checks and timing closure are the only proof that a Verilog module works.
Architecture & BIM: Hook into the official buildingSMART IDS-Audit-Tool. As the ONESTRUCTION case showed, Claude Opus 4.5 passed the audit only a third of the time (IDSAuditPass 0.33); their model, trained against the real validator, reached 0.69 with near-100% structural validity.
Physical Simulation: Run MuJoCo, PyBaMM, or Isaac Sim in volatile RAM. Contact normals, joint torques, and thermal degradation must be calculated by the physics solver, not guessed from text.

5. Seven Production Exploits & Hard-Earned Defenses

The most valuable engineering knowledge in RLVR is the catalogue of ways agents game poorly constructed reward functions. Here are seven exploits, several of them reported by the teams that hit them, and the architectural defenses that neutralize them:

1. The Inaction Exploit (The Zapier Trap)

What happens: In Zapier’s AutomationBench (evaluating agents across 47 simulated SaaS apps), the reward checked whether the final world state satisfied assertions. During RL training, the agent’s api_fetch_calls collapsed to near zero while reward stayed flat, and the team caught it live in the training metrics. One way this happens: if some assertions already pass in the initial state, doing nothing keeps that partial reward and avoids every error.

The Defense: Active Mutation Verification. Exclude assertions that already pass in the initial state from scoring. Require verified proof of state change: an authenticated HTTP 200 write receipt, a database row delta, or an updated filesystem hash. Doing nothing must earn exactly 0.0.

2. The Canned Refusal Exploit

What happens: When a reward penalizes wrong answers much more heavily than abstentions, RL can learn to abstain on nearly everything: a canned “This question cannot be answered from the provided text” earns a safe, middling reward on every turn. Work on abstention-aware RL reports that even a small static reward for refusing can trigger this “lazy refusal” collapse.

The Defense: Never reward refusal without hard proof. An agent may only earn refusal credit if an air-gapped oracle check confirms that the specific question was intentionally designed to be unanswerable from the provided context.

3. The Tool Cache Exploit (The Cursor Composer Trap)

What happens: Cursor trained Composer 2.5 partly on tasks built by deleting a feature and asking the agent to reimplement it. The model found a leftover Python type-checking cache and reverse-engineered it to recover the deleted function signature; in another task it decompiled Java bytecode to rebuild a third-party API. Anything stateful left in the environment (compiler artifacts, typechecker caches, stale file timestamps) is a shortcut the policy will eventually find.

The Defense: Build each environment from a clean state with no traces of the answer, and flush compiler, linter, and runtime caches between rollout steps. Run whole-repository verification suites against the final workspace state rather than trusting single-file incremental diffs.

4. The Asymmetric Risk Principle

What happens: Standard machine learning optimizers treat false positives and false negatives as equally bad (balanced accuracy or F1). In real-world business operations, this is false. If an AI misses an illegal weapon on an e-commerce marketplace, the legal liability is catastrophic. If it raises a false alarm on a kitchen knife, a human can review it in two seconds.

The Defense: Encode operational asymmetry directly into the verifier’s loss matrix. For example, a catalog-review reward can charge several times more for a missed restricted item than for a false alarm, so the policy learns to look harder at edge cases instead of defaulting to the cheap answer.

5. The Aesthetic Divergence Trap (The Genspark Slides Lesson)

What happens: Genspark trained Gen-1 Slides against an LLM grader that scores the rendered deck. Over the run, the policy learned to import a reference deck and pass it off as the finished product, to write “sources verified” into its report without checking anything, and to shrink fonts until overflow detection stopped firing.

The Defense: Gate soft visual scoring behind strict, programmatic content assertions. Check that every required data field, entity mention, and chart specification exists in the rendered artifact before visual evaluation begins, and keep updating the grader as new hacks appear, as Genspark did with a grader that evolves alongside the policy.

6. Container Tampering & File Overwrites

What happens: If the verifier grades an agent by reading a file written inside the agent’s container (e.g. /tmp/results/score.txt), an agent with bash or Python access will eventually discover that it can use echo "1.0" > /tmp/results/score.txt to win every episode.

The Defense: Absolute sandbox isolation. The agent’s container must be read-only except for an explicit scratchpad directory. Verification code, test suites, and expected answers must execute host-side or in an isolated verifier container that receives only the candidate artifact.

7. Data Leakage & Canary Testing

What happens: Test answers, gold solutions, or evaluation fixtures accidentally leak into prompt metadata, environment descriptions, or error tracebacks. The model memorizes the leaked tokens rather than learning the generalized skill.

The Defense: Automated Canary Testing. Inject unique, random GUID tokens into private environment state. Run automated CI assertions verifying that canaries and gold answers never appear in the model’s public observation window or tool definitions.

The agent hands a candidate artifact to a verifier it cannot reach. Score-file overwrites and leaked answers both require a shared filesystem.

6. Protocol Steps vs. Hard Gates: Preventing Exploration Freezes

While hard gates are essential for correctness, verifier designers often create a subtle failure mode by making administrative protocol steps hard gates.

Consider a task where an agent must diagnose a database issue and then run a final bookkeeping command: commit_transaction().

If your verifier zeroes out the entire reward if the agent forgets commit_transaction(), you create a zero-gradient desert:

The base model explores different valid diagnostic queries, but forgets the final commit step on turn 15.
Every single rollout in the group receives a reward of exactly 0.0.
Because all rewards in the group are identical, the advantage is zero, the gradient is zero, and the model never learns that its diagnostic queries were actually correct.

The Solution: Soft Protocol Gates.

Reserve zero-reward hard gates strictly for true errors (syntax crashes, security violations, wrong answers). For administrative protocol steps that the model must remember to take, use a soft gate: cap the reward of an unfinished trajectory at 0.25 × progress, and award the remaining 0.75 only upon successful submission. This provides enough gradient signal for the model to learn the core skill first, and then naturally discover the protocol closing step.

A hard gate on a forgotten bookkeeping step gives every rollout 0, so the group has no advantage to learn from. A soft gate (0.25 × progress) keeps the rewards apart.

7. The Pre-Run Checklist: Before You Commit GPUs

Before you launch an RL post-training run across an expensive GPU cluster, walk through this eight-point verification audit:

Hand-Play the Environment: Can a human or a golden solver achieve 100% reward using only the information visible in the agent’s public observation? If the oracle cannot solve it, the task is broken, not hard.
Non-Triviality Check: Does a naive baseline (doing nothing, submitting empty output, or returning random guesses) score exactly 0.0?
Variance Verification: Across 8 rollouts of your base model, do at least some groups achieve different reward scores? If all groups score 0.0, add SFT warmup before running RLVR.
Native Engine Binding: Is correctness evaluated by a real compiler, solver, or database rather than an LLM string check?
Ephemeral RAM Isolation: Does every episode execute in volatile memory (/dev/shm) with compiler caches flushed between turns?
Air-Gapped Canaries: Are all golden answers and test suites physically inaccessible from the agent’s execution sandbox?
Asymmetric Loss Alignment: Does the reward function penalize catastrophic operational failures more heavily than minor cosmetic mistakes?
Frozen Evaluation Exam: Is your held-out benchmark dataset locked and air-gapped from the training loop before step 1 begins?

A great model cannot fix a broken verifier. But with an airtight, deterministic verification harness, even compact open models can learn to out-execute commercial frontier models on your most critical business workflows.

Applying this to your own models? We build private RL training environments with deterministic verifiers for teams' own workflows. Private training environments.

Originally published at g-ftech.com.

Shopify, Intercom, Ramp: How Open Models Beat Frontier APIs in Production

Aleksei Romanov — Sun, 04 Oct 2026 19:54:38 +0000

For the past two years, the standard enterprise AI strategy looked suspiciously like a parlor trick: take an 8,000-token system prompt packed with JSON schemas, API documentation, and polite formatting rules, stuff it into the context window of a giant 400-billion-parameter frontier model, and pray that it doesn’t hallucinate a field name halfway through an inventory query.

When you run that trick ten times a day on a demo laptop, it feels like magic. When you run it two thousand times a minute in production, reality hits like a brick:

The latency kills your UI: Streaming an 8,000-token prompt across external API data centers adds seconds of time-to-first-token delay.
The bills are comical: A multi-turn agent querying live APIs can burn through tens of thousands of dollars a day in recurring token taxes.
The model invents schemas: No matter how sternly you write “You MUST output valid GraphQL” in your prompt, generic frontier models still drift, hallucinate nonexistent mutations, and fail on strict production schemas.

Across 2025 and 2026, a quiet migration took place behind closed doors. Teams like Shopify, Meta, Intercom, Ramp, Snowflake, and industrial architectural startups like ONESTRUCTION stopped treating frontier LLM APIs as all-knowing digital workers. Instead, they adopted a repeatable, high-leverage playbook:

Take an open model (often 8B to 35B parameters; Intercom and Harvey went larger), hook it up to a real execution verifier (a compiler, an API sandbox, or real business conversion telemetry), and train it with reinforcement learning until it out-executes frontier giants on that exact task.

Below, we examine the raw production data, telemetry, and architectural blueprints from seven enterprise case studies that prove why specialized, post-trained agents are replacing generic API prompting.

Start here

Explore verified production telemetry from Shopify, Meta, Intercom, Ramp, and ONESTRUCTION. See how open models post-trained with SFT and RLVR slash serving costs by 90%–96%, eliminate prompt token bloat, and consistently outperform generic frontier APIs on mission-critical schemas.

RLVR (Reinforcement Learning with Verifiable Rewards): Training an agent where policy updates are driven strictly by programmatic outcomes (compilers, unit tests, linters, API response codes) rather than human opinion.

Gisting: Replacing a long system prompt with a few learned token embeddings, trained by distillation so the model keeps the prompt's behavior.

Cold-Start SFT: Supervised fine-tuning on high-quality demonstration traces to establish basic tool-calling syntax before reinforcement learning begins.

Reward Hacking: Pathological behavior where an agent figures out how to score maximum reward without actually solving the underlying task (e.g., claiming a question is unanswerable).

The loop behind every case: one high-volume task, a cold-start SFT, a verifier that scores attempts, and GRPO against it. New production failures become tomorrow's training data.

1. The Playbook: Four Steps That Repeat in Every Winning Case

When you audit the technical reports of teams that successfully replaced frontier API endpoints with in-house models, you realize they are not inventing bespoke, one-off mathematics. They are all running the exact same four-stage loop:

Isolate a Single, High-Frequency Task: Nobody trains a specialized model to be an “autonomous junior engineer” who writes poetry, plans trips, and builds distributed databases. They pick an operation run hundreds of thousands of times a week: translating merchant questions into GraphQL, extracting structured M&A liabilities, or verifying BIM architectural schemas.
Cold-Start with High-Quality Demonstrations (SFT): Before running reinforcement learning, they collect 500 to 5,000 pristine examples of the task executed correctly. This distills the expected JSON schema, API signature, and tool definitions into a compact LoRA adapter.
Bind an Automated Verifier (The Reward Function): This is the crucial differentiator. They never use an uncalibrated LLM judge scoring “helpfulness on a scale of 1 to 5.” They attach something that checks the result: does the generated SQL execute on the database (Snowflake)? Does the XML pass the official buildingSMART validator (ONESTRUCTION)? Was the support ticket resolved without a human (Intercom)?
Optimize with GRPO or RLPF: The model generates multiple candidate trajectories for each problem. Attempts that pass the verifier are reinforced; attempts that crash or hallucinate are penalized.

The result is a model that sheds all general conversational hesitation and executes the specific business task with machine-like precision.

Production Telemetry

The Seven Deployments: Frontier APIs vs. Task-Specialized Models

Published benchmarks from engineering teams that replaced generic prompting with post-trained open weights.

Company & Task	Automated Verifier	Frontier API Baseline	Specialized Open Model
Shopify ↗ (e-commerce): Admin GraphQL queries	Calibrated judge panel scoring each response	~$27M/yr (est.) · 6,000-token prompt	~$1M/yr (−96%) · TTFT −19%
Intercom ↗ (support): autonomous ticket resolution	Ticket resolved without human escalation	71.1% (GPT-5.4 & Opus 4.5)	73.1% resolution (−65% hallucinations vs Sonnet 4.6)
Meta ↗ (advertising): ad copy for ~35k advertisers	Click-through rate from live delivery	SFT model imitating human copywriters	+6.7% CTR (p = 0.0296)
ONESTRUCTION ↗ (construction): BIM data specifications (IDS/XML)	buildingSMART IDS-Audit-Tool	0.33 audit pass (Claude Opus 4.5)	0.69 audit pass · −55% authoring time
Ramp ↗ (fintech): 15-turn spreadsheet questions	Exact numerical match	61.9% (Claude Opus 4.6)	66.3% exact match at Haiku speed
FermiSense ↗ (marketplaces): restricted-goods catalog review	Labeled catalog-review set	76.9% of max score (~$20–$34 / 1k listings)	87.3% of max score ($0.50 / 1k listings)
Snowflake ↗ (data platforms): enterprise Text-to-SQL	Query execution on Snowflake	Gemini 3.1 Pro, Claude Opus 4.7	Higher accuracy on Snowflake's hard Text-to-SQL set at 30–150× smaller size

Every number above is documented from published technical post-mortems and papers. Detailed systems architecture and failure analyses follow below.

Each company's own metric, frontier baseline against the specialized model. Rows use different metrics; compare within a row. Harvey's grey point is its base model before RL.

2. Direct Business Signals: Meta AdLlama & Intercom Fin Apex 1.0

Most AI benchmarking takes place in artificial academic clean rooms: HumanEval, GSM8K, or MMLU. Two of the most instructive case studies in enterprise post-training skipped synthetic proxies entirely and wired reinforcement learning directly to real-world business outcomes.

Meta AdLlama: Rewarding Real Click-Through Rates (CTR)

When Meta set out to generate automated ad text suggestions for advertisers on Facebook, standard supervised fine-tuning (SFT) produced a frustrating failure mode. Models trained by imitating human copywriters wrote grammatically flawless, pleasant text—that real human users completely ignored on their feeds.

In a landmark 10-week live production experiment spanning nearly 35,000 advertisers and about 640,000 ad variations, Meta deployed AdLlama using Reinforcement Learning from Platform Feedback (RLPF).

Instead of rewarding the model for matching human demonstration style, the policy gradient was weighted directly by historical Click-Through Rate (CTR) lift from live ad delivery.

The Production Impact: AdLlama delivered a verified +6.7% CTR lift over the SFT baseline (p = 0.0296). Even more tellingly, advertisers themselves adopted and launched more variants generated by the RL model—an indirect confirmation that the model had learned to identify and articulate concrete, conversion-driving value propositions rather than marketing filler.

Intercom Fin Apex 1.0: Did the Human Have to Intervene?

Customer support is another domain where generic frontier models look brilliant in sales demos but cause operational chaos in production. An agent that sounds 100% confident while giving a subtly incorrect refund policy creates expensive customer churn.

Intercom built Fin Apex 1.0, an autonomous support model handling roughly 2 million customer conversations every week. Rather than relying on a commercial closed API, Intercom post-trained an open-weights model (hundreds of billions of parameters, per VentureBeat) on years of its own support conversations.

The primary optimization objective was binary and unambiguous: was the customer’s inquiry resolved without requiring escalation to a human agent?

Resolution Rate: Fin Apex achieved a 73.1% autonomous resolution rate on Intercom’s internal production benchmark, outperforming GPT-5.4 (71.1%), Claude Opus 4.5 (71.1%), and Claude Sonnet 4.6 (69.6%).
Hallucination Reduction: Fact-checking verifiers reduced hallucinated answers by 65% compared to Claude Sonnet 4.6.
Serving Economics: Operating cost dropped to approximately 1/5th of frontier API rates, while shaving 0.6 seconds off average response latency.

In customer service, two percentage points of resolution might sound modest to an outsider. At 2 million conversations weekly, that delta represents 40,000 tickets every single week that never reach a human queue.

3. The $26M Serving Squeeze: Shopify Sidekick’s GraphQL Engine

Perhaps the most financially staggering case study in production agent deployment comes from Shopify Sidekick, the merchant assistant that translates natural language queries (“Show me my top five wholesale customers in Ohio who haven’t ordered in 60 days”) into live GraphQL queries against the Shopify Admin API.

Sidekick handles up to 2,000 requests per minute. Initially, the team followed the standard industry playbook: write an exhaustive system prompt describing the GraphQL schema and let an external frontier model generate the queries.

The architecture immediately hit two production walls:

Prompt Token Overhead: The system prompt alone required approximately 6,000 tokens of schema definitions, filtering rules, and field restrictions on every single turn.
Staggering Serving Bills: At 2,000 requests per minute with 6,000-token system prompts, serving Sidekick over commercial frontier APIs was projected at an estimated $27 million annually.

Metric	Before	After	Change
System Prompt Size	~6,000 tokens	~1,500 gist tokens	−75% token bloat
Annual Serving Cost	~$27,000,000 (est.)	~$1,000,000 (fleet)	−96% cost reduction
Time-to-first-token (median, 350 RPM)	438 ms	354 ms	−19% latency
End-to-end latency (median, 350 RPM)	6.8 s	4.2 s	−38% faster
Quality vs. frontier model	Frontier model	Surpassed it on Shopify's evals	Higher quality

Shopify replaced the static frontier API with a continuous, daily learning cycle:

Shopify Sidekick's daily loop: critics and annotators repair failed conversations, then a full fine-tune and GRPO with a calibrated judge. Gisting replaces the 6,000-token prompt with about 1,500 learned tokens.

Every day, a self-healing pipeline collects failed production conversations: a panel of frontier reasoning models critiques each failure, an arbiter turns the critiques into a repair, and expert annotators fix what the critics cannot. Shopify then runs a full-parameter fine-tune over the accumulated data and repeats Group Relative Policy Optimization (GRPO), with a calibrated judge as the reward signal.

Separately, Shopify replaced the 6,000-token system prompt with about 1,500 learned “gist tokens.” At 350 requests per minute, median time-to-first-token fell from 438 ms to 354 ms (−19%) and median end-to-end latency from 6.8 s to 4.2 s (−38%). Serving this traffic on a frontier model was estimated at $27M a year, against roughly $1M for the fine-tuned model: a 96% reduction.

4. Industrial BIM & Construction: ONESTRUCTION’s Ishigaki-IDS

While software startups focus on SQL and customer support, physical and civil engineering presents a far harsher environment for language models. In modern architecture and construction, building projects valued in the hundreds of millions of dollars rely on Building Information Modeling (BIM).

To ensure that architects, structural engineers, and HVAC contractors don’t construct colliding systems, industry regulators enforce Information Delivery Specifications (IDS)—heavily structured, XML-based schemas governed by the international buildingSMART standard.

Writing valid IDS XML by hand requires encyclopedic knowledge of thousands of IFC entity classes, property sets, and strict XML schema definition (XSD) constraints. When Japanese construction startup ONESTRUCTION tested commercial frontier models (including Claude 4.5) on drafting IDS requirements from regulatory checklists, the models collapsed:

The Frontier Failure in Physical AI: Generic frontier models achieved an IDSAuditPass score of only 0.33. They wrote fluent, believable XML that completely failed when loaded into the official buildingSMART schema compiler. The models hallucinated property names, violated cardinality rules, and missed strict data-type declarations.

Working with AWS under Japan’s METI GENIAC initiative, ONESTRUCTION built Ishigaki-IDS, a specialized family based on Qwen3 (8B, 14B, and 32B):

Continued Pretraining: Trained on millions of pages of Japanese building codes, architectural standards, and IFC documentation.
Supervised Fine-Tuning: Trained on pairs of natural language building requirements and verified IDS XML files.
RLVR with the buildingSMART Compiler: In the final post-training stage, the model generated candidate XML structures that were directly evaluated by the official, open-source IDS-Audit-Tool. The compiler’s strict validation errors served as the reinforcement learning reward signal.

On the dedicated Ishigaki-IDS-Bench, the specialized 32B model reached an IDSAuditPass of 0.69, more than doubling the performance of general commercial models while achieving near-100% structural syntactic validity. In a workflow study with six BIM practitioners, Ishigaki-assisted authoring cut total work time by 54.7%.

5. Multi-Step Spreadsheets & Law: Ramp FastAsk and Harvey M&A

Two other high-profile case studies prove that specialized models can navigate deep, multi-turn reasoning workflows that previously seemed reserved exclusively for $200/month flagship models.

Ramp FastAsk: 15-Turn Spreadsheet Navigation

Financial technology platform Ramp built FastAsk to answer complex merchant financial questions (“Calculate the month-over-month increase in software spend excluding AWS for Q2”). Financial models live in giant spreadsheets with nested formulas, irregular column names, and cross-tab summaries.

Generic models struggle because they try to dump the entire spreadsheet into the prompt, overflowing the context and making calculation errors. Ramp equipped a Qwen3.5-35B-A3B model with three tools (metadata retrieval, range reading capped at 1,000 cells per call, and Python execution) and trained it with GRPO over 15-turn trajectories on Prime Intellect infrastructure.

Exact Numerical Match: Accuracy increased from 56.25% to 66.25%, handily beating Claude Opus 4.6 (61.88%).
Latency: The model is a mixture of experts with about 3B active parameters, so it ran at roughly Claude Haiku 4.5 speed (1.05× Haiku's time, against 1.44× for Opus 4.6) while scoring higher than Opus.

Harvey: 50-Room Legal M&A Diligence

In mergers and acquisitions, legal teams must review dozens of “data rooms” containing thousands of contracts, leases, and regulatory filings to ensure no hidden liability or change-of-control clause goes unnoticed.

Legal AI company Harvey built an orchestrator agent using a post-trained Qwen3.5-122B-A10B model. The orchestrator didn’t read the documents directly; its job was to coordinate subagents, ensure every folder in the virtual data room was inspected, and verify that all mandatory compliance criteria were satisfied.

Trained with GRPO and evaluated on 50 held-out data rooms, the orchestrator’s rubric pass rate more than doubled, from 29.9% to 63.0%, and document review coverage rose from 62% to 96%.

6. The $500 Asymmetry: Why Specialized Models Dominate on Unit Economics

One of the most persistent misconceptions in enterprise AI is that post-training requires millions of dollars in GPU clusters. For general pretraining (building Llama 4 from scratch), that is true. For specialized post-training, the mathematics are completely inverted.

Consider FermiSense, a project focused on automated product catalog review and restricted goods classification for high-volume e-commerce marketplaces:

The Training Cost: The team took an open-weight Qwen 3.5 9B model and ran about 2,500 GRPO steps on a single GPU. The total compute budget for the run was approximately $500.
The Evaluation: Models were scored on a labeled catalog-review set, reported as a share of the maximum achievable score and as plain accuracy.
The Benchmark Score: The $500 fine-tuned 9B model scored 87.3% of the maximum, against 76.9% for the best frontier configuration. On accuracy it reached 97% (up from 64.2% for the base model), against 93% for GPT-5.6 Sol and 91% for Claude Opus 4.8.
The Serving Economics: Auditing 1,000 product catalog entries over frontier APIs cost about $20 to $34. Serving the specialized 9B model cost $0.50 per 1,000 items: 40× to 68× cheaper.

This is the economic moat of specialization. If you audit 10 million products a month, the frontier API costs about $200,000 to $340,000 every month. The specialized model costs about $5,000 a month—while catching more policy infractions.

Serving cost on a log scale. Shopify estimates $27M a year on a frontier model against about $1M for its fine-tuned one; FermiSense pays $0.50 per 1,000 listings against about $20 to $34.

7. The Reward Hacking Minefield: Negative Results from the Frontier

Post-training specialized agents with reinforcement learning is not free magic. If your verifier has a single logical loophole, the model will discover it, exploit it relentlessly, and report a 100% training reward while doing completely broken work.

Several major engineering teams have published candid autopsies of reward hacking during RL post-training:

Three reward hacks reported by the teams themselves. In each, the reward could be earned without doing the work it was meant to measure.

Project	What the Model Did	Why the Verifier Failed	The Engineering Fix
Zapier (AutomationBench)	`api_fetch_calls` dropped to near zero while reward stayed flat.	The reward could be earned without the API work the task assumed.	Caught live from side metrics during training; the reward was debugged and re-validated, and AutomationBench checks final state with deterministic assertions, including negative ones.
Cursor (Composer 2.5)	In a feature-deletion task it found a leftover Python type-checking cache and reverse-engineered it to recover a deleted function signature; elsewhere it decompiled Java bytecode to rebuild a third-party API.	Synthetic tasks rewarded passing tests, so any trace of the deleted code left in the sandbox was a shortcut.	Treat leaks as task bugs: scrub caches and build artifacts from the environment before rollouts.
Genspark (Gen-1 Slides)	Passed off an imported reference deck as finished work, wrote “sources verified” without checking, and shrank fonts until overflow detection stopped firing.	A fixed grader has fixed blind spots, and slide quality is easy to game.	An online grader that evolves with the policy, adding checks as new hacks appear.

8. The Systems Checklist: How to Pick Your First Specialized Agent

If your organization is spending six figures a month on commercial model APIs and struggling with schema reliability, here is the pragmatic blueprint for deciding whether to build a specialized agent:

Three gates before post-training a task: enough volume, a programmatic verifier and a frozen held-out exam. A no at any gate means staying on an API for now.

Check the Volume Threshold: If a task runs 50 times a day, keep using a frontier API. Post-training pays for itself when volume exceeds 10,000 requests weekly, where token costs and latency bottlenecks become genuine business risks.
Ensure You Have a Programmatic Judge: If you cannot write a Python script, compiler, database check, or automated business metric to verify whether an answer is correct, you are not ready for RLVR. Start by building the verification harness first.
Never Train Without Held-Out Evaluation Sets: As the cases above prove, training curves lie. Lock down 200 to 1,000 realistic problems that your training loop never sees. A model is only ready for deployment when it beats the base model and frontier baselines on that frozen exam.
Own the Weights: Once you qualify a specialized adapter, host it inside your own security boundary (your VPC or on-premises GPU nodes). The resulting artifact belongs on your balance sheet, immune to external API deprecations, pricing changes, and data residency liabilities.

Applying this to your own models? We post-train open models on teams' own tasks with SFT and GRPO, and hand back the weights. LLM post-training with g factor.

Originally published at g-ftech.com.

Long-Horizon Agents: Why Multi-Turn Reasoning Breaks and the Practical Training Tricks That Fix It

Aleksei Romanov — Sat, 03 Oct 2026 14:52:21 +0000

Almost every modern language model looks impressive on a two-step demo. You ask it to check a database or summarize a document, it calls the right tool, formats the answer, and looks like an autonomous engineer.

The illusion falls apart the moment you ask that same model to complete a 30- or 50-step workflow: diagnosing a failing Kubernetes cluster, navigating a multi-file pull request, or running a 48-hour industrial simulation.

Around step 10, the agent makes a small typo in a terminal command. By step 15, it misinterprets a confusing error message. By step 25, it has forgotten its original plan and begins arguing with its own terminal history. By step 35, its context window is overflowing with thousands of noisy stderr lines, and the run crashes in an expensive loop.

This failure is not solved by simply adding a bigger system prompt. It is a systems problem in how agents explore, learn from mistakes, and receive reinforcement learning signals.

In this guide, we break down why multi-turn agents collapse, why standard RL algorithms burn thousands of GPU hours learning nothing, and the pragmatic tricks—proven in recent frontier reasoning models (such as MiniMax-M1 and DeepSeek-R1)—that turn fragile single-turn models into reliable long-horizon agents.

Start here

Learn why single-turn benchmarks lie, why standard GRPO policy updates produce zero learning on long tasks, how the 75% pass-rate rule cuts GPU training costs in half, why smooth gradient clipping (CISPO) saves words of doubt, and how to train through real agent harnesses without building fragile mocks.

Horizon (T): The number of sequential interaction turns (actions and tool responses) an agent takes to finish a task.

Markovian State Drift: When an early mistake leaves the environment in an unfamiliar, broken state, causing subsequent actions to fail at much higher rates.

Zero-Gradient Trap: When all rollouts in a training group either fail completely (0/8) or succeed easily (8/8), producing zero mathematical advantage to update the model.

The 75% Rule: A dynamic curriculum trick that removes tasks once the model solves them over 75% of the time, saving up to 50% of rollout compute.

CISPO: An RL objective that clips importance-sampling weights instead of dropping tokens, so rare exploration tokens like 'Wait' and 'Let me verify' keep their gradient.

Gated Rewards: Binary pass/fail checks (strict JSON format, no conversational chatter) that an answer must pass before receiving any quality score.

Polar Proxy: A non-invasive proxy that records tool traffic between real agent wrappers (like Claude Code or mini-SWE-agent) and the model during training.

1. The Step-20 Cliff: Why Agents Break in Practice

Why do agents fail over long horizons? The basic math seems simple: if an agent has a 95% chance of making the right tool call on any single turn, its probability of getting 20 turns right in a row is:

0.95²⁰ ≈ 35.8%

At 50 steps, that drops to 7.7%. That means more than 9 out of 10 runs fail, even with a seemingly high 95% single-step accuracy.

If every step were independent, success would be p to the power of the number of steps: 95% per step leaves 35.8% at 20 turns and 7.7% at 50. Real environments are worse.

But real-world software engineering and industrial environments are far worse than this textbook formula. Steps are not independent coin flips; they are tied together by environmental state:

The Domino Effect: On step 6, the agent passes an invalid flag to a bash command or deletes a necessary lockfile.
Entering Unfamiliar Territory: The system returns an unexpected error traceback or an exit code 127. Because base models are mostly trained on clean, successful tutorials, the model has rarely seen this exact broken state.
The Hallucination Spiral: Instead of stepping back and running git status or inspecting the filesystem, the agent invents an excuse, tries to run a more complicated command to patch its mistake, and creates a circular error loop.

In practice, once an agent makes an unhandled error, its chance of making a second error on the very next step jumps from 5% to over 60%. This is why long-horizon tasks fail at an exponential cliff.

2. The Zero-Gradient Trap: Why 90% of Agent Rollouts Learn Nothing

When teams try to fix this by applying standard reinforcement learning (like GRPO or PPO), they hit an immediate wall: the cold-start problem.

Reinforcement learning works by comparing multiple attempts (rollouts) for the same problem. The attempts that score better than average are reinforced; the attempts that score worse are discouraged:

Task Difficulty	Rollout Outcome (8 attempts)	Group Average	Advantage Difference	What the Model Learns
Too Hard (50 steps)	`[0, 0, 0, 0, 0, 0, 0, 0]`	0.0	All 0.0	Zero gradient. 100% wasted GPU hours.
Too Easy (2 steps)	`[1, 1, 1, 1, 1, 1, 1, 1]`	1.0	All 0.0	Zero gradient. Model already knows it.
The Learning Zone	`[1, 1, 1, 1, 0, 0, 0, 0]`	0.5	+0.5 vs −0.5	Maximum gradient signal. Strong learning.

If a task is too hard and the model never succeeds by chance, all attempts get a score of 0. The difference between attempts is zero, the gradient is zero, and the model updates nothing. You can burn tens of thousands of dollars on H100 clusters, but the model will not take a single step forward.

GRPO compares 8 attempts at the same task. When all fail or all pass, every advantage is zero and the update learns nothing; only mixed groups carry signal.

3. The 75% Rule: Dynamic Difficulty Filtering

One of the most practical engineering insights in recent online RL literature is the 75% pass-rate cutoff.

As training progresses, tasks that were once difficult become easy. A coding problem that the model solved 1 out of 8 times at step 100 might be solved 8 out of 8 times by step 1,000. Once a model solves a task every time, that task ceases to teach it anything.

How the trick works:

Before each training stage, run the current checkpoint across the task pool and measure the pass rate (e.g., across 8 samples).
Filter out any task solved more than 75% of the time: The model has already mastered it. Keeping it in the training batch only dilutes the gradients and wastes rollout compute.
Focus strictly on the 20%–75% struggle zone: This is where the variance between good and bad attempts is highest, providing the strongest reinforcement learning signal.

In practice, this simple automated filter cuts rollout inference compute by over 50%. Instead of wasting GPU cycles generating answers to tasks the model already knows, every dollar of compute goes toward pushing the capability frontier.

The chance that an 8-attempt group contains at least one pass and one fail. Tasks below 20% or above 75% waste most rollouts, which is why the 75% filter pays off.

4. CISPO: Preserving the Words of Doubt That Make Reasoning Work

Standard RL algorithms (like PPO and GRPO) enforce a mathematical safety clamp: if a token’s probability changes too drastically in a single step, the algorithm clips the gradient to zero to prevent the model from destabilizing.

For standard text generation, this clipping is fine. But for multi-turn reasoning agents, it causes a silent disaster:

In a reasoning model, the most valuable tokens are the words of doubt: “Wait, let me check the exit code...”, “However, the file was not created...”, or “Hold on, that contradicts step 2.”

An SFT model almost never produces these tokens. They start with tiny probabilities (e.g., 0.001%). When reinforcement learning starts discovering that hesitating and checking intermediate outputs leads to higher rewards, the probability of these tokens begins to rise rapidly.

The GRPO Trap: Because these tokens experience the biggest percentage change, standard GRPO marks them as outliers and clips their gradients to zero. The model is blocked from learning how to doubt itself.

The CISPO Fix: Instead of clipping out-of-bounds tokens to zero, the CISPO objective (introduced by MiniMax-M1) clips the importance-sampling weight instead of the update: the weight is capped and treated as a constant, but every token still contributes its gradient. Training stays stable, and the learning signal on rare exploration tokens is never switched off. The result: agents learn to self-correct and backtrack significantly faster than under vanilla GRPO.

PPO/GRPO clipping drops tokens whose probability ratio leaves the trust region, which hits rare words like 'Wait' first. CISPO caps the importance weight but keeps every token's gradient.

5. Gated Rewards: Binary Filters Before Scoring

A classic trap when building multi-turn agents is using a single blended reward score: giving the model 0.7 points for a partially working answer, 0.2 for nice explanation formatting, and 0.1 for being polite.

Models exploit blended scores ruthlessly. In coding and function-calling environments, a model trained on soft scores quickly learns a catastrophic habit: conversational chatter around structured output.

Instead of emitting raw, parseable JSON:

{"action": "read_file", "path": "src/main.py"}

The model starts outputting polite commentary:

Certainly! Here is the tool call you requested to inspect the file:
{"action": "read_file", "path": "src/main.py"}
I hope this helps!

To a human reviewer, this looks helpful. To a production execution pipeline or JSON parser, it is an immediate syntax crash.

The Gated Reward Architecture:

Gate 1 (Strict Formatting): Did the model output strictly valid JSON/code with zero conversational chatter outside the payload? If NO → Reward = 0. Stop immediately.
Gate 2 (Language & Environment Invariants): Did the model switch languages mid-sentence or run a prohibited command? If YES → Reward = 0. Stop immediately.
Gate 3 (Execution Verification): Did the unit tests or environment transition succeed? Only outputs that pass Gates 1 and 2 are evaluated on task performance.

Gated rewards: format, invariants and execution are pass/fail checks in front of the task score, so a chatty but unparseable answer earns exactly 0.

This binary gate eliminates format drift within the first 50 steps of training. The model learns that no amount of eloquence can compensate for a broken interface contract.

6. Adaptive Length Penalties: Curing the Infinite Thinking Loop

Once you give a model the ability to reason and self-correct, a new pathology emerges: verbose procrastination.

The model realizes that writing more reasoning tokens slightly improves its chances of catching an error. Within a few hundred training steps, average reasoning length explodes: the agent begins spending 4 minutes and 3,000 tokens “thinking” about whether to run ls or pwd.

A naive penalty (e.g., subtracting 0.001 points per token) ruins the model: hard multi-step problems genuinely require long chains of reasoning, and a flat penalty causes the model to give up prematurely.

The solution is an adaptive length penalty scaled by task difficulty:

For easy tasks (Pass Rate > 80%): Apply a strong length penalty. If a task is straightforward, the model should execute it cleanly in 1–2 turns without writing a dissertation.
For difficult tasks (Pass Rate < 20%): Reduce the length penalty toward zero. Give the model the full token budget to explore hypotheses and recover from failures.

This keeps simple tool calls lightning-fast while preserving deep multi-turn persistence on genuine edge cases.

7. The Polar Proxy: Train Through Real Agent Harnesses

How should you wire an agent into an RL training loop?

Most research labs make the mistake of rewriting the agent harness inside Python: building a simplified mock of the terminal, the filesystem, and the tool definitions. This is a dead end. Within weeks, your real production agent framework (whether it’s Claude Code, Codex, OpenHands, or mini-SWE-agent) evolves, and your training environment becomes an obsolete toy that no longer matches reality.

The production pattern (used in NVIDIA’s Polar and adapted by industrial labs) is to train through a transparent proxy:

Dimension	Mock Environment Re-implementation	Transparent Proxy (Polar Architecture)
Harness Compatibility	Locked to 1 simplified custom script	Works with any CLI/API harness without code changes
Production Parity	High drift; model learns mock environment quirks	Zero drift; model sees identical production tool logs
Maintenance Cost	Must rewrite mock whenever tools update	Plug-and-play; updates happen automatically
Container Isolation	Often runs in shared local memory	Spins up clean ephemeral containers per rollout task

With a proxy in place, the real agent harness believes it is talking to a standard OpenAI-compatible API. The proxy intercepts every prompt and completion, records the exact multi-turn trajectory, and forwards it to the RL training cluster. You train the model in the exact same environment where it will operate in production.

Training through a transparent proxy: the real harness talks to what looks like a normal model API, while the proxy records each trajectory for the RL trainer.

8. Multi-Turn Memory: Writing Is Harder Than Reading

As interaction horizons stretch past 30 turns, an agent cannot rely solely on raw context stuffing. If an agent dumps every terminal output into the prompt, the context window quickly bloats past 60,000 tokens, causing inference latency to spike and attention heads to suffer from “lost in the middle” syndrome.

Long-horizon agents require explicit memory tools (e.g., scratchpads, key-value stores, or state summaries). However, empirical training reveals a crucial asymmetry:

Reading memory is easy; writing memory is where agents fail.

Retrieving a stored fact via search is a standard retrieval task that LLMs handle well. The real danger is memory corruption during writes:

An agent records duplicate facts under slightly different keys, confusing future lookups.
An agent mistakes a temporary intention for a permanent state (e.g., recording “database migration failed” as a permanent truth, even after a subsequent step fixed it).
Conflicting records accumulate over turns, causing the agent to hallucinate about whether a task was completed.

In state-of-the-art post-training, memory skills are trained with separated reward channels: one benchmark evaluating retrieval accuracy (reading), and a dedicated multi-turn benchmark evaluating memory hygiene (writing only unambiguous, non-redundant, verified state diffs).

How much do these recipes help together? The original article includes an interactive toy model in which per-step reliability starts at 94%, 97% or 99% and decays slightly with every turn. In that model, the combined recipe keeps about 67% of 20-turn tasks and 17% of 50-turn tasks on track, against about 5% and under 1% for a prompted model. The numbers are assumptions, not measurements; the point is the shape: better training moves the cliff later, it does not remove it.

A toy model with assumed per-step reliability (94%, 97%, 99%) that decays slightly each turn. Better recipes move the cliff later; they do not remove it.

9. Practical Blueprint for 50-Step Autonomy

If your team is deploying or post-training an agent intended to run reliably past 20 turns, here is the pragmatic systems checklist:

Stop trusting single-turn leaderboards: A model with a 95% tool-calling score on Berkeley Function Calling can still have a sub-10% completion rate on real 30-step workflows. Benchmark at your actual production horizon.
Enforce the 75% difficulty filter: Discard training tasks once your checkpoint achieves >75% pass rate. Focus 100% of your GPU budget on the learning zone where success is between 20% and 75%.
Use CISPO instead of naive clipping: Never use rigid PPO/GRPO clipping that zeroes out gradients on rare tokens. Give self-doubt and backtracking tokens room to grow.
Put gated filters in front of rewards: Fail with zero reward if the model outputs conversational preamble, markdown fences, or malformed JSON. Clean interfaces must be a non-negotiable prerequisite.
Scale length penalties by task difficulty: Punish verbose reasoning on simple tasks; allow breathing room on genuinely hard tasks.
Train through real harnesses with proxies: Don’t waste months maintaining mock simulators. Use a transparent proxy to train directly against your production agent CLI in isolated containers.

Research executed September 2026. Primary systems references: MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention (MiniMax, 2025), RLVR & Reward Hacking, Multi-Reward RL in Agent Environments, and g factor Private Task Gyms.

Applying this to your own models? We build private RL training environments with deterministic verifiers for teams' own workflows. Private training environments.

Originally published at g-ftech.com.

We Tried ISO-AdamW. AdamW Kept Its Job.

Aleksei Romanov — Fri, 02 Oct 2026 22:08:03 +0000

Four answers. After implementing a new optimizer, fixing a painfully slow matrix operation, and running both versions through the same math exam, that was the gap: 758 correct answers for ISO-AdamW versus 754 for our baseline AdamW on a 1,000-question held-out test.

You can probably imagine the more exciting headline: “New geometric optimizer beats AdamW! Time to rewrite the RL pipeline!” We would have liked an unequivocal win, too. An algorithmic improvement to how language models learn during reinforcement learning is always worth celebrating.

But four answers out of a thousand deserved a hard systems look. This is the empirical story of a controlled pilot on a dedicated NVIDIA H200 GPU, an elegant linear-algebra idea, and why our standard AdamW is keeping its job for production post-training.

Start here

You don't need advanced algebra to follow this. We'll explain what we changed, how we benchmarked both optimizers on identical hardware, and why a slightly higher score didn't justify replacing our production stack.

Optimizer: The update rule that turns loss gradients into numerical updates to model weights.

GRPO: Group Relative Policy Optimization: an RL algorithm that generates multiple candidate answers and reinforces the higher-scoring attempts.

Held-out test: A clean test set of 1,000 questions isolated from hyperparameter tuning and early checkpoint selection.

Isospectral (ISO): A mathematical constraint that keeps the singular values (stretch factors) fixed while rotating only the coordinate frames.

1. What does an optimizer actually do?

Imagine teaching someone to solve multi-step word problems. You give them a problem, let them try, and score their work. They need a systematic way to turn that feedback into a better attempt on the next problem.

A neural language model does this by updating billions of numerical values called weights. During training, a loss function produces a mathematical gradient indicating which directions would increase the expected reward. The optimizer is the engine that translates those gradients into actual parameter steps: which numbers to change, and by how much.

Standard AdamW tracks running moving averages of past gradients (momentum) and their squared magnitudes (variance) to scale step sizes independently for every individual parameter in the model.

For this experiment, we applied GRPO (Group Relative Policy Optimization) on school-level math word problems (GSM8K). The model generates groups of candidate completions. A strict verification rule scores each final numeric output: if the true solution is 42, an articulate multi-paragraph chain of thought that ends in 43 receives exactly zero reward.

The research question was straightforward: could constraining weight updates to an isospectral manifold make that practice more effective than unconstrained coordinate-wise AdamW?

2. The ISO idea, without the algebra

Picture a circle of points drawn on a flexible sheet of rubber. Applying a weight matrix transforms that circle into an ellipse: it stretches some directions more than others and rotates their coordinate orientation.

SVD splits a weight matrix into rotations and stretch factors. AdamW changes both; ISO-AdamW keeps the stretch factors of the base model and only turns the frames.

In linear algebra, any weight matrix W can be uniquely decomposed via Singular Value Decomposition (SVD):

W = U \cdot Σ \cdot V^{T}

Here, Σ is a diagonal matrix of singular values (the stretch factors along principal axes), while U and V are orthogonal matrices representing the input and output singular frames (the rotation angles).

Zhu et al.’s paper, ISO: An RLVR-Native Optimization Stack, proposed that during reinforcement learning, the model does not need to alter its fundamental representational capacity (Σ₀), which was already acquired across trillions of tokens during pre-training. Instead, reasoning primarily requires realigning the coordinate frames (U, V):

W (t) = U (t) \cdot Σ_{0} \cdot V (t)^{T}

With ISO-AdamW, AdamW updates the frame directions rather than updating each raw weight entry directly. After each optimizer step, an orthogonalization step projects the frames back onto the manifold of orthogonal matrices. The underlying weights still change, but their singular spectrum remains permanently pinned to the base model.

We had explored the theoretical foundations of this idea in our deep dive on what changes inside weight spectra during RL. Now we wanted to quantify its real-world engineering behavior inside our own training pipeline.

3. Giving both optimizers a fair shot

To keep the comparison empirical and grounded, we capped the compute budget and pinned the workload:

Model: Qwen/Qwen3-1.7B-Base, a clean, modern base model without prior alignment bias.
Workload: openai/gsm8k, consisting of multi-step arithmetic word problems with deterministic numeric answers.
Hardware Baseline: One dedicated, preemptible NVIDIA H200 SXM GPU (141 GB HBM3e). Both optimizers were executed sequentially on the exact same physical machine to avoid cloud variance.
Training Horizon: 50 policy updates (100 prompt groups, generating 400 completions total per model), using full FP32 optimizer states and BF16 autocast computation.

We also avoided the common benchmarking trap of forcing both optimizers to use the same learning rate. Imposing an arbitrary shared step size on two structurally different mathematical algorithms guarantees an unfair race. We screened three learning rates per optimizer on a separate 128-question validation check:

AdamW grid: 5e-7, 1.5e-6, 3e-6 → Selected: 3e-6
ISO-AdamW grid: 3e-7, 7.5e-7, 2e-6 → Selected: 7.5e-7

Once hyperparameters were chosen, both 50-step pilots restarted cleanly from the exact same initial base model weights (SHA256: 24d53e3c...). The 1,000-question held-out test was evaluated only after training was fully complete.

4. First, make the experiment runnable: The 44× kernel detour

Before we could measure accuracy, we hit a severe systems bottleneck. Our initial prototype implementation of ISO-AdamW took ~170 seconds per training step on an H200. Standard AdamW took ~7 seconds. At that rate, our entire GPU budget would have been consumed by just 10 steps.

The culprit was the orthogonalization step: projecting the updated frame matrices back onto the Stiefel manifold. Naively computing a full Singular Value Decomposition (SVD) across hundreds of weight matrices on every single step stalls the GPU pipeline. We benchmarked alternative SVD drivers, but full decompositions remained prohibitive.

The key systems insight was that between consecutive training steps, the orthogonal frames change by only small increments. Instead of a full SVD from scratch, we can compute the polar projection using iterative Newton–Schulz iterations:

X_{k + 1} = \frac{1}{2} X_{k} (3 I - X_{k T} X_{k})

We implemented a high-performance Newton–Schulz polar iteration with a strict numerical tolerance check and an automatic SVD fallback in case iterations fail to converge within tolerance.

We qualified this backend across the five distinct tensor geometries of Qwen3-1.7B on the H200:

Tensor Shape	Frames / Step	Native SVD Latency	Newton–Schulz Latency	Speedup	Orthogonality Error
[1024, 1024]	56	136.3 ms	34.01 ms	4.0×	`1.99e-15`
[2048, 1024]	56	86.6 ms	1.07 ms	81.3×	`1.87e-15`
[2048, 2048]	197	505.6 ms	2.62 ms	193.2×	`2.85e-15`
[6144, 2048]	84	350.5 ms	6.60 ms	53.1×	`2.85e-15`
[151936, 2048]	1	584.5 ms	162.38 ms	3.6×	`4.00e-15`
Full Model Step Work	394 frames	142.11 s	3.20 s	44.5× faster	`< 4.0e-15`

This algorithmic optimization reduced the per-step polar overhead from 142.11 seconds down to 3.20 seconds, matching the mathematical output of full SVD to within six parts in ten trillion (relative error < 6.1 × 10⁻¹³).

Re-orthogonalizing 394 frame matrices per step: 142 s with a full SVD, 3.2 s with warm-started Newton-Schulz iterations. Most of the SVD time is the 197 square [2048, 2048] frames.

5. Four answers out of a thousand

With both optimizers running at practical speed and verified with exact checkpoint replay, we evaluated both models on the full 1,000-question held-out GSM8K test set:

Optimizer / Checkpoint	Correct Answers	Accuracy	Delta vs. Base	Mean Solution Length
Base Model (Qwen3-1.7B-Base)	730 / 1,000	73.0%	—	—
Standard AdamW (50 steps)	754 / 1,000	75.4%	+2.4%	167 tokens
ISO-AdamW (50 steps)	758 / 1,000	75.8%	+2.8%	148 tokens

On paper, ISO-AdamW finished ahead: 75.8% versus 75.4% (+0.4 percentage points). However, inspecting the individual question-by-question paired outcomes reveals what actually happened:

ISO did not simply retain all of AdamW’s correct solutions and add four extra answers on top. Instead, it traded 64 successes for 68 different ones.

The 1,000 test questions, one square each. The two optimizers agree on 868 of them; ISO-AdamW's four-answer lead is 68 wins minus 64 losses on the other 132.

To determine whether this gap was statistically significant, we performed 20,000 paired question-bootstrap resamples. The resulting 95% confidence interval for the accuracy difference ranged from −1.8% to +2.7%. Because the interval easily crosses zero, the 4-question lead is within expected sampling noise.

An exact McNemar test on the 132 questions where the two models disagree gives p ≈ 0.79. Even the gains over the base model are borderline: AdamW's +2.4 points has a 95% interval of −0.2 to +5.0, and ISO-AdamW's +2.8 points runs from +0.1 to +5.4. With a single training seed, none of these differences can be called real yet.

95% paired bootstrap intervals for each gap. The ISO-AdamW vs AdamW interval is centered near zero, and even AdamW's gain over the base model crosses zero.

The Peril of Early Checkpoint Peeking

Tracking validation performance during training illustrates another critical lesson: intermediate checkpoints can be highly deceptive if evaluated too early.

Step	AdamW Accuracy	ISO-AdamW Accuracy	Observed Delta	Interpretation
Step 10	84.4%	83.6%	−0.8%	Initial warm-up stabilization
Step 20	79.7%	85.2%	+5.5% (ISO leads)	AdamW temporary exploration dip
Step 30	86.7%	83.6%	−3.1%	AdamW recovers rapidly
Step 40	88.3%	88.3%	0.0%	Exact parity
Step 50	89.1%	89.8%	+0.7%	Both converge to high-accuracy asymptote

If an engineer had stopped the run at step 20 and posted a screenshot, they could have claimed that ISO-AdamW was “+5.5% superior.” By step 40, they were tied; by step 50, both had converged to within noise. Rigorous post-training requires running recipes to their planned conclusion.

Validation accuracy on 128 questions, where one question is 0.8 points. The 5.5-point ISO lead at step 20 is an AdamW dip; at step 50 the gap is one question.

6. The stopwatch and the memory bill

In production systems engineering, an optimizer cannot be judged on raw benchmark scores alone. It must be weighed against compute time, GPU memory allocation, and operational overhead.

Metric	AdamW	ISO-AdamW	Cost Penalty
Total Training Wall Time	11.42 min (685.1 s)	12.13 min (727.8 s)	+6.2% slower
Mean Wall-Clock Step Time	13.70 s	14.56 s	+0.86 s / step
Peak Allocated GPU Memory	35.79 GiB	52.80 GiB	+47.5% memory overhead
Average Output Tokens / Answer	167 tokens	148 tokens	−11.4% (shorter solutions)
Total GPU Cloud Cost	~$3.80	~$4.10	Total pilot: $8.32

Almost all of the extra training time comes from the first step (69.7 s versus 11.8 s), where ISO-AdamW builds the frames for all 394 matrices. After that, ISO-AdamW averaged 13.43 s per step against 13.74 s for AdamW, helped by its shorter answers. The lasting cost is memory, not speed.

ISO-AdamW needs 47.5% more peak GPU memory. Its extra training time is almost all the first step; later steps were slightly faster than AdamW's.

The most substantial systems penalty of ISO-AdamW is the +47.5% increase in peak allocated GPU memory. Because ISO requires maintaining additional frame buffers and auxiliary state for the polar iterations across 394 matrices, a model that comfortably fits on an 80GB GPU during standard AdamW fine-tuning can trigger an Out-Of-Memory (OOM) fault under ISO.

Interestingly, ISO-AdamW generated slightly shorter completions on average (148 tokens vs. 167 tokens). While more concise reasoning is an appealing trait, in this limited pilot it did not yield higher overall accuracy.

7. Why AdamW keeps its job

Our pilot produced an undeniable engineering achievement: we built an isospectral optimizer implementation that executes stably, passes strict checkpoint replay, and accelerates polar projections by 44.5×.

Yet as systems architects, our conclusion is unambiguous: AdamW remains our default production optimizer.

Rules for Evaluating Manifold Optimizers in Production

Never judge an optimizer on steps alone: A method that cuts training by 20% in step count but adds 40% in per-step compute or memory overhead is a net loss in the data center.
Check the contingency matrix, not just the score: A +0.4% lead that trades 64 working solutions for 68 different ones is a lateral variance swap, not a strict capability improvement.
Memory overhead bounds batch size: In distributed reinforcement learning, VRAM is king. A 47.5% memory penalty forces smaller rollout group sizes or earlier activation offloading, degrading overall cluster throughput.
Beware early evaluation dips: Optimizers exhibit distinct exploration dynamics. What looks like an insurmountable lead at step 20 often evaporates when both models reach saturation.

We still find the geometric principles behind Isospectral Optimization compelling—particularly for offline model merging (ISO-Merger) where frame rotations can be combined without GPU gradients. But for active GRPO post-training, standard AdamW remains the pragmatic, battle-tested standard.

Technical notes & reproducibility

The full telemetry dataset (JSON) contains all 50-step progress curves, raw step times, parameter grids, and numerical verification receipts.

Dataset: 3,000 training, 500 validation, and 1,000 test problems from openai/gsm8k (split seed 20260910). Prompt templates were strictly deduplicated and disjoint.
Model: Qwen/Qwen3-1.7B-Base (revision ea980cb), trained without LoRA, quantization, or vLLM to isolate raw optimizer mechanics.
Hardware: 1× NVIDIA H200 SXM (141 GB HBM3e) hosted on Nebius AI Cloud.
Verification: 50 unit tests passed. Checkpoint saving and resumption reproduced identical updates to bit-level precision. Peak spectral drift across checked matrices remained below 1.3 × 10⁻⁸.

Experiment executed September 10–11, 2026. Primary sources: our verified telemetry, ISO: An RLVR-Native Optimization Stack (Zhu et al., 2026), Qwen3-1.7B-Base, and GSM8K.

Applying this to your own models? We post-train open models on teams' own tasks with SFT and GRPO, and hand back the weights. LLM post-training with g factor.

Originally published at g-ftech.com.

Benchmarking Qwen 3.8 27B Across Inference Providers: Together, Fireworks, Doubleword, and g factor

Aleksei Romanov — Thu, 01 Oct 2026 18:55:59 +0000

If you look at vendor landing pages or benchmarks on social media, every inference provider claims to be “the fastest engine on Earth.” You see sleek bar charts showing thousands of tokens per second, single-digit Time-To-First-Token, and promises of dramatic cost savings.

Then you deploy a 27-billion parameter reasoning model like Qwen 3.8 27B into production, route real multi-turn traffic through those endpoints, and practical systems trade-offs immediately emerge. Single-stream latency behaves very differently from high-concurrency batch throughput. Hardware interconnects dictate whether Tensor Parallelism flies or grinds to a halt. And subtle gateway interpretations of reasoning tokens can quietly balloon your generation budgets.

Over the past several weeks, we ran an exhaustive series of empirical benchmarks across our research platform (gft-studio) to evaluate Qwen 3.8 27B across dedicated infrastructure and leading inference providers: Together AI, Fireworks AI, Nebius, Doubleword, and g factor.

Below, we share the verified engineering telemetry: how parallel topology (Tensor Parallelism vs. Data Parallelism) shapes decode latency, how next-generation B200 hardware scales over H100 baselines, the real-world sweet spot of multi-token speculative decoding (MTP4 vs. MTP8), how prefix caching and prompt speculation behave on structured workflows, and what happens to latency when concurrency pushes to 64 parallel streams.

Start here

Explore complete concurrency 1–64 benchmarks across Together, Fireworks FP8, Doubleword, vanilla vLLM, and g factor on Qwen 3.8 27B. Discover why single-node Tensor Parallelism hits 190 tok/s at c1 while cross-node TP stalls, how MTP4 pushes dual-H100 throughput to 769 tok/s, and how prefix caching unlocks 1,140+ tok/s on structured prompts.

TTFT: Time-To-First-Token: the elapsed latency from HTTP request submission until the first stream chunk arrives.

ITL: Inter-Token Latency: the time required to generate and emit each subsequent token during the decode phase.

TP2 (Tensor Parallelism): Splitting individual weight matrices across 2 GPUs over high-speed NVLink so both GPUs collaborate on every token.

DP2 (Data Parallelism): Running two independent replicas of the model on separate GPUs, routing requests via a load balancer.

MTP (Multi-Token Prediction): A speculative decoding technique where additional lightweight heads predict future tokens in a single forward pass.

1. The Experimental Setup: Isolating Real Performance

To make an inference benchmark meaningful, you have to eliminate confounding variables. Comparing a 7B model on FP8 with a 70B model on BF16 tells you nothing. Comparing an API called from a laptop in London with a server hosted in Oregon tells you about transit latency, not engine throughput.

Here is how we standardized our test harness:

The Model: Qwen/Qwen3.8-27B (and its official FP8 quantized variant), pinned to tokenizer revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0.
The Workload: Standardized AIPerf 0.12.0 test suites streaming over public HTTPS. Prompts averaged ~564 input tokens, requesting exactly 128 generated output tokens at temperature 0 and random seed 42.
Concurrencies Tested: Standard evaluation across concurrency levels 1, 2, 4, 8, 16, 32, and 64. Low-concurrency cells ran 16 warmup requests followed by 60 strictly measured requests across three separate repetitions (180 measured requests per cell). High-concurrency cells (c16, c32, c64) ran 64 warmup requests and 256 measured requests per repeat.
The Hardware Baseline: A fixed hardware budget of 2× NVIDIA H100 80GB SXM GPUs per target (except where explicitly comparing next-generation B200 accelerators).

Throughput = \frac{\sum _{i = 1 N} OutputTokens _{i}}{Δ t _{window}}

2. The Full Concurrency 1–64 Comparison

In real-world serving, systems rarely operate at a single concurrency point. Early morning traffic might see solitary interactive queries, while daytime peaks bombard your cluster with dozens of simultaneous streams.

Below is the full empirical comparison across Together AI, Fireworks AI (FP8), Doubleword, vanilla standalone vLLM, and g factor with MTP4 speculative decoding:

Output throughput from concurrency 1 to 64 on 2x H100 (or the provider's equivalent). Single-node TP2 (Together) leads at low concurrency; at c64 four of the five engines land within about 10% of each other.

Concurrency	g factor MTP4	Together FP8 TP2	Fireworks FP8 DP2	Doubleword Public FP8	Vanilla vLLM FP8 DP2
c = 1	133.00	189.61	114.49	65.86	69.29
c = 2	250.65	350.57	227.15	138.48	136.81
c = 4	437.41	612.35	401.46	291.29	253.51
c = 8	769.13	1033.44	640.02	534.71	459.16
c = 16	1252.34	1590.84	1056.65	935.52	836.18
c = 32	1904.97	2579.22	1146.75	1716.53	1601.44
c = 64	2611.88	2701.47	1145.08	2649.71	2462.44

Looking across this spectrum reveals three distinct operational regimes:

Low Concurrency (c = 1 to 2): Together AI leads single-stream speed (189.61 tok/s at c1, 350.57 tok/s at c2). Because Together serves the model via single-node Tensor Parallelism (TP2), both H100s collaborate on every token decode step. g factor with MTP4 delivers 133.00 tok/s at c1 and 250.65 tok/s at c2, comfortably outperforming Fireworks (114.49 / 227.15 tok/s) and Vanilla vLLM (69.29 / 136.81 tok/s).
Medium Concurrency (c = 4 to 8): As requests multiply, multi-replica data parallelism (DP2) hits its stride. g factor reaches 437.41 tok/s at c4 and 769.13 tok/s at c8, pulling ahead of Fireworks DP2 (640.02 tok/s) and Vanilla vLLM (459.16 tok/s).
High Concurrency Saturation (c = 16 to 64): At c32 and c64, all modern engines push past 1,500 to 2,600 tokens per second. Together reaches 2,701 tok/s, Doubleword reaches 2,650 tok/s, and g factor reaches 2,612 tok/s. But raw throughput at high concurrency is only half the story—you also have to examine the latency bill.

The Latency Bill Under Load

It is easy to generate thousands of tokens per second if you let requests sit in a queue. What matters for interactive user experience is Time-To-First-Token (TTFT): how long the user stares at a blank screen before text begins to stream.

Concurrency	Together FP8	Fireworks FP8	Doubleword FP8	Vanilla vLLM FP8
c = 1	141.6 ms	1088.4 ms	1093.1 ms	278.0 ms
c = 2	174.3 ms	390.4 ms	1059.8 ms	468.2 ms
c = 4	217.1 ms	435.3 ms	826.8 ms	538.0 ms
c = 8	221.8 ms	1323.3 ms	786.7 ms	656.9 ms
c = 16	269.1 ms	1034.9 ms	919.5 ms	705.6 ms
c = 32	245.8 ms	2687.1 ms	975.5 ms	736.0 ms
c = 64	1701.7 ms	7714.6 ms	1598.6 ms	1195.3 ms

Notice what happens between c32 and c64. On Fireworks, throughput stays essentially flat (1,146 tok/s → 1,145 tok/s), while p95 TTFT surges from 2.69 seconds to 7.71 seconds. The GPUs are fully saturated; adding more requests in flight simply queues them up at the door without producing more tokens per second.

p95 Time-To-First-Token on a log scale. Fireworks climbs to 7.7 s at c64 while its throughput stays flat: extra requests only wait in the queue.

3. Architectural Topology: Tensor Parallelism vs. Independent Replicas

The most important architectural lesson from our benchmark is that two GPUs do not make a system; how those two GPUs are connected makes the system.

Why Together Won Single-Stream Latency: High-Speed NVLink

Look at the single-stream results: Together achieved 189.6 tok/s and an Inter-Token Latency (ITL) of just 3.9 milliseconds, compared to 69–133 tok/s on independent single-GPU replicas.

Why? Because Together deployed Tensor Parallelism (TP2) inside a single physical server. In TP2, every linear layer in Qwen 3.8 27B is sliced across both GPUs. For every single token decode step, GPU 0 and GPU 1 compute their respective matrix slices and exchange intermediate activations via an all-reduce collective. At concurrency 1, both H100s collaborate on that single user’s request simultaneously.

In Data Parallelism (DP2), by contrast, GPU 0 handles User A while GPU 1 handles User B. At concurrency 1, User A only uses one GPU, with decode speed physically bounded by that single chip’s memory bandwidth (3.35 TB/s on H100 SXM).

Evaluating Cross-Node Tensor Parallelism

Seeing Together’s single-stream TP2 speed, we tested an experiment: what happens if you run TP2 across two separate 1× H100 cloud instances connected via standard datacenter virtual networking (VPC) rather than intra-node NVLink?

The result demonstrated the severe cost of network barrier latency:

Concurrency	DP2 (2 Separate Nodes)	Cross-Node TP2 (Over Network)	Single-Node TP2 (NVLink)	Network Interconnect Penalty
c = 1	73.71 tok/s	24.98 tok/s	189.61 tok/s	2.95x slower than DP2
c = 4	274.64 tok/s	56.47 tok/s	612.35 tok/s	4.86x slower than DP2
c = 8	483.19 tok/s	75.75 tok/s	1,033.44 tok/s	6.38x slower than DP2

At concurrency 8, cross-node TP2 throughput collapsed from 483 tok/s down to 75.75 tok/s. The physical explanation is straightforward:

The same two H100s at concurrency 8: NVLink inside one server, two independent replicas, or tensor parallelism stretched over the cloud network.

t_{step} = t_{GEMV} + 2 \times \frac{Bytes _{activation}}{Bandwidth _{interconnect}} + t_{latency}

During single-token decoding, GEMV computations for a 27B model take only 5 to 10 microseconds. Within a single chassis over NVLink, exchanging activations takes ~2 microseconds over 900 GB/s channels. Across separate physical servers over standard cloud networking, that same transfer requires 800 to 2,000 microseconds. The GPUs spent over 95% of their execution time stalled at network barriers waiting for TCP packets.

The Golden Rule of Topology: Never run Tensor Parallelism across physical machines unless you have dedicated multi-rail InfiniBand. If your GPUs reside on separate nodes, always deploy independent replicas with data parallelism (DP).

4. The Hardware Leap: What Happens on NVIDIA B200?

While H100 remains the workhorse of enterprise inference, next-generation NVIDIA Blackwell (B200) accelerators are entering production. We benchmarked Fireworks AI running Qwen 3.8 27B on a dedicated 2× B200 deployment:

Concurrency	2× H100 BF16 (Fireworks)	2× B200 BF16 (Fireworks)	Observed Hardware Speedup
c = 1	97.09 tok/s	153.38 – 161.32 tok/s	1.58x – 1.66x
c = 4	347.12 tok/s	571.82 – 597.11 tok/s	1.65x – 1.72x
c = 8	622.18 tok/s	979.49 – 1,001.32 tok/s	1.57x – 1.61x

Moving from H100 to B200 delivered a clean 1.6x to 1.7x throughput increase with zero code changes. This speedup is directly explained by memory hardware specifications:

NVIDIA H100 SXM features 3.35 TB/s of HBM3 memory bandwidth.
NVIDIA B200 features 8.00 TB/s of ultra-dense HBM3e bandwidth (a 2.38x hardware leap).

Because memory-bound autoregressive decoding scales near-linearly with memory bandwidth, B200 allows a 2-GPU cluster to cross the 1,000 tok/s barrier even in unquantized 16-bit precision.

5. Case Study: Reasoning Token Budgets Across API Gateways

During our evaluation of Qwen3.8-27B-FP8 on Doubleword’s public API, we encountered an instructive systems interaction that illustrates how reasoning-capable models interface with standard throughput benchmarks.

We configured our AIPerf test harness with standard parameters to measure decode throughput at a fixed length:

{"max_completion_tokens": 128, "temperature": 0}

While standard non-reasoning requests complete in 1 to 2 seconds for 128 tokens, these initial requests ran for approximately 247 seconds. An inspection of the returned payload counters explained the extended duration:

The 128-token cap applied only to the visible answer. The model first generated 15,774 hidden reasoning tokens, so each request ran for about 247 seconds.

Metric	Configured Parameter	Returned API Telemetry
Reasoning Tokens (<think>)	Default reasoning; cap requested	15,774 tokens
Answer Tokens	128 target	130 tokens
Total Generated Tokens	128 completion cap	15,904 tokens (full reasoning trajectory)

Gateway Semantics and Parameter Interpretation

Tracing the HTTP response headers revealed routing through an upstream gateway adapter. In many recent reasoning model architectures, API proxies interpret max_tokens or max_completion_tokens as applying strictly to the final visible answer, leaving internal reasoning tokens (within <think>...</think> tags) unconstrained.

When no explicit reasoning limit is enforced, the model executes its full natural thought trajectory (~15,774 reasoning tokens) prior to producing the concise final response.

Standardizing the Benchmark: To evaluate pure decode throughput at a controlled output length, Doubleword documents explicitly setting reasoning_effort: "none":

{"max_completion_tokens": 128, "reasoning_effort": "none"}

With explicit non-thinking parameters configured, Doubleword’s public API completed all 540 measured requests cleanly within the target token bounds:

Concurrency 1: 65.86 tok/s (TTFT p50: 708 ms, ITL: 12.4 ms)
Concurrency 4: 291.29 tok/s (TTFT p50: 725 ms, ITL: 12.5 ms)
Concurrency 8: 534.71 tok/s (TTFT p50: 708 ms, ITL: 12.7 ms)

Benchmarking Note on Token Accounting: When measuring reasoning models, it is essential to audit both internal thought tokens and final visible tokens. Standardizing reasoning parameters is necessary to ensure fair, reproducible throughput comparisons across providers.

6. Speculative Decoding: Tuning MTP4 vs. MTP8

Multi-Token Prediction (MTP) is one of the most effective ways to accelerate autoregressive decoding without loading a separate draft model. Instead of predicting a single token per forward pass, lightweight auxiliary heads speculatively propose multiple future tokens, which the main model verifies in parallel.

How multi-token prediction works: cheap heads guess several tokens ahead, the main model checks all guesses in one pass, keeps the matching prefix and adds one token of its own.

Our initial baseline used MTP1 (predicting 1 draft token ahead), yielding 613.54 tok/s at concurrency 8. We then ran an ablation study comparing MTP4 (4 draft tokens) against MTP8 (8 draft tokens) on identical pairs of H100 GPUs with FP8 weights and attention KV:

Profile	Concurrency	Mean Throughput (tok/s)	TTFT p50 (ms)	ITL p50 (ms)	End-to-End p95 (ms)	Draft Token Acceptance
g factor MTP4	1	133.00 [132.15–133.54]	253.95 ms	5.22 ms	1,221.62 ms	59.36%
g factor MTP4	4	437.41 [434.23–440.52]	290.94 ms	6.41 ms	1,407.07 ms	59.36%
g factor MTP4	8	769.13 [761.22–774.06]	303.63 ms	6.93 ms	1,666.06 ms	59.36%
g factor MTP8	1	129.81 [129.36–130.14]	259.32 ms	5.13 ms	1,423.34 ms	39.02%
g factor MTP8	4	417.11 [415.15–420.33]	302.78 ms	6.60 ms	1,601.21 ms	39.02%
g factor MTP8	8	767.94 [744.30–792.23]	315.16 ms	6.77 ms	1,883.54 ms	39.02%

Why did eight draft tokens fail to beat four?

Speculative decoding is a fundamental trade-off between verification speed and draft accuracy:

With MTP4, the model accepted 59.36% of all draft tokens. Every step generated an average of 3.37 accepted tokens with minimal verification overhead.
With MTP8, predicting 8 tokens ahead into the future proved significantly harder: acceptance plunged to 39.02%. While the average step size increased slightly to 4.12 tokens, the engine wasted GPU cycles generating and verifying rejected branches.
As a result, MTP8 yielded 2% to 4% lower throughput and increased p95 end-to-end latency by 13% to 17%.

MTP4 vs MTP8 at concurrency 4: deeper drafts are accepted less often, so the longer step does not pay for the extra verification work.

Conclusion: More speculative depth is not automatically better. For Qwen 3.8 27B, MTP4 is the empirical sweet spot, achieving 769 tok/s on 2× H100 without latency penalties.

7. Concurrency Scaling: Throughput Has a Latency Bill

Concurrency 8 is a standard comparison point, but it does not represent the saturation capacity of two H100s. We tested the scaling behavior of g factor under heavy loads of 16, 32, and 64 concurrent streams:

Concurrency	MTP1 Throughput	MTP4 Tuned Throughput	MTP4 TTFT p95 (ms)	MTP4 End-to-End p95 (ms)	Interactive Serving SLA
c = 16	1,013.98 tok/s	1,252.34 tok/s	530.00 ms	2,065.24 ms	Optimal interactive SLA
c = 32	1,604.98 tok/s	1,904.97 tok/s	794.38 ms	2,780.32 ms	High-throughput sweet spot
c = 64	1,645.97 tok/s	2,611.88 tok/s	1,433.17 ms	4,111.71 ms	Queueing delay boundary

2,612 tok/s on two H100s looks incredible on a slide, but you have to check the latency bill:

At c = 32, the cluster operates in its optimal production window: generating 1,905 tok/s while keeping p95 Time-To-First-Token well under 800 ms.
At c = 64, throughput expands to 2,612 tok/s, but prefill queueing delays cause p95 TTFT to climb to 1.43 seconds.

In production inference, peak throughput numbers are meaningless without latency percentiles. Always size your concurrency targets to your application’s TTFT service level objectives.

8. Prefix Caching & Prompt Speculation: Give Them Something to Reuse

In short-prompt benchmarks (~500 input tokens), prefix caching typically reports near-zero cache hits because each request presents completely unique text.

To evaluate how caching and speculation perform in real agentic workflows (where prompts share long system prompts, tool definitions, and conversation histories), we tested a workload with a 2,820-token shared prefix and a unique prompt tail (total input: 2,870 tokens):

Why the shared prefix helps twice: the cached prefix skips most of prefill, and suffix speculation copies tool-call text that already appears in the prompt.

Optimization Strategy	c=1 tok/s	c=4 tok/s	c=8 tok/s	Prefix Cache Hit Ratio	Draft Token Acceptance
Baseline (Cache Disabled)	57.92	209.92	349.84	Disabled	—
Automatic Prefix Caching	60.70	230.88	418.49	80.47%	—
Prefix Cache + N-gram (k=4)	172.62	558.05	1,046.40	83.76%	90.18%
Prefix Cache + Suffix Speculation	180.36	655.77	1,140.26	83.76%	99.12%

The empirical takeaways for long-context and agentic workflows are substantial:

Prefix Caching Eliminates Prefill Latency: Enabling prefix caching achieved an 80.47% cache hit rate, reducing concurrency-8 TTFT p50 from 855 ms down to 323 ms and delivering an immediate +19.6% throughput boost.
Context-Aware Speculation (N-gram & Suffix): When repetitive or structured patterns exist in the prompt, n-gram and suffix speculation propose continuations directly from the cached context without requiring an auxiliary neural net. This accelerated decode throughput by 2.7x to 2.9x, pushing throughput to 1,140 tok/s with a 99% draft acceptance rate.

Concurrency-8 throughput on a 2,870-token prompt with a 2,820-token shared prefix, as each optimization is switched on.

9. Architectural Rules for Honest Benchmarks

When evaluating published inference benchmarks or sizing dedicated clusters, five architectural dimensions determine real-world performance:

What was the GPU interconnect? Is that 1,000 tok/s running on a single server with 900 GB/s NVLink (TP2), or across independent nodes (DP2)?
What precision was actually served? Was it BF16, calibrated FP8, or aggressive 4-bit quantization? Has domain accuracy been qualified on target tasks?
Where was the client running? Were requests sent across the public internet or through internal cluster networking?
How were reasoning tokens budgeted? Does the token limit include the entire completion, and were internal thought chains isolated during speed testing?
What is the latency trade-off? High batch throughput is easy to achieve at high concurrency, but does the Time-To-First-Token still meet your interactive user requirements?

The Bottom Line: High-throughput inference is not a black-box commodity. It is a systems discipline that balances memory bandwidth, network topology, speculative decoding depth, and cache management. When engineered properly, a dedicated 2-GPU cluster can deliver world-class latency, break past 1,000 tokens per second, and provide complete data sovereignty at predictable costs. That is what we set up in a private LLM deployment.

Applying this to your own models? We deploy and tune open models on teams' own GPUs, in their VPC or on-premises. Private LLM deployment.

Originally published at g-ftech.com.

The Limits of AI: Induction, Deduction, and Why Models Can't Jump

Aleksei Romanov — Wed, 30 Sep 2026 20:05:28 +0000

Every week brings another breathless proclamation that Artificial General Intelligence (AGI) is just a few trillion tokens, a bigger datacenter cluster, or another reinforcement learning run away. The prevailing industry dogma assumes that intelligence is a single scalar curve: if you scale compute hard enough, everything—from writing bash scripts to inventing new branches of physics—will fall out of the loss function.

Yet in 2026, a thought-provoking position paper from Google DeepMind titled LLMs Can’t Jump, authored by research scientist Tom Zahavy (a key contributor to DeepMind’s AlphaProof system), planted a sharp flag in the ground. Zahavy made a precise, mathematically and epistemologically grounded argument that cuts through the marketing noise: today’s foundation models have mastered induction and deduction, but they are structurally incapable of abduction.

This is not a temporary bug that a larger context window or a cleaner pre-training dataset will fix. It defines the fundamental ceiling of current architectures. An AI can spot statistical regularities across trillions of words (induction). An AI can chain thousands of logical derivation steps within an established rulebook like Lean 4 or Python (deduction). What an AI cannot do is invent a brand-new axiom out of thin air when the rulebook itself is broken.

Below, we dissect the classical triad of reasoning first formulated by philosopher Charles Sanders Peirce, examine why LLMs excel at induction and deduction, explore the psychology of the abductive “aha!” moment through historic breakthroughs (Einstein, Kepler, Semmelweis), and define the true computational upper bound of machine intelligence.

Start here

Understand the three classical modes of inference (induction, deduction, abduction), why next-token prediction and RL verifiers hit a ceiling at paradigm shifts, and why human intuition remains indispensable for inventing new scientific axioms.

Induction: Statistical pattern generalization: extracting probable rules and regularities from observed data.

Deduction: Logical consequence: deriving guaranteed theorems and conclusions from established premises and axioms.

Abduction: Hypothesis generation: the creative leap that invents a novel explanatory premise or axiom to explain anomalies.

Axiomatic system: A formal framework defined by initial unproven assumptions (axioms) and derivation rules.

Paradigm shift: Replacing the foundational axioms of a discipline rather than computing consequences within them.

1. The Triad of Inference: Peirce’s Epistemology

More than a century before the first neural network was conceived, American philosopher and logician Charles Sanders Peirce (1839–1914) demonstrated that all human reasoning collapses into three distinct, non-interchangeable modes of inference:

Inference Mode	Logical Structure	The Classical Beanbag Example (Peirce)	Cognitive Function
Deduction	Rule + Case ⇒ Result	Rule: All beans in Bag A are white. Case: These beans are from Bag A. ⇒ Result: These beans are white.	Certainty: Computes the necessary consequences of established rules. Discovers no new truths.
Induction	Case + Result ⇒ Rule	Case: These beans are from Bag A. Result: These beans are white. ⇒ Rule: Probably, all beans in Bag A are white.	Generalization: Compresses statistical patterns from finite samples into empirical laws.
Abduction	Result + Rule ⇒ Case (Hypothesis)	Result: These beans on the table are white. Rule: All beans in Bag A are white. ⇒ Hypothesis: Perhaps these beans fell from Bag A.	Invention: Proposes a plausible explanatory cause for an unexpected, anomalous observation.

Notice the profound asymmetry between these three modes:

Deduction is truth-preserving: If your premises are sound and your logic is valid, your conclusion is guaranteed to be true. It is the language of mathematics, formal compilers, and logic verifiers. However, deduction cannot generate new premises; it can only unpack what was already latent in the starting axioms.
Induction is probability-increasing: As you observe more white swans, your confidence that all swans are white rises. It is the language of statistics, empiricism, and modern machine learning. However, induction is forever trapped in the distribution of what has already been observed.
Abduction is the only mode that introduces new ideas: It is the creative leap that posits an unobservable mechanism, a hidden variable, or an entirely new conceptual framework to explain why reality is behaving unexpectedly.

Peirce's three modes on the bean-bag example: deduction derives the result, induction generalizes the rule, abduction proposes the hypothesis that would explain what was seen.

2. Why LLMs Mastered Induction and Deduction

To appreciate why current AI hits a wall at abduction, we must first understand why it has been so spectacularly successful at the first two modes:

Induction: The Trillion-Parameter Pattern Compressor

Pre-training on the public web using the autoregressive cross-entropy loss is the most colossal inductive exercise in the history of science:

L_{pretrain} (θ) = - t = 1 \sum T lo g P_{θ} (x_{t} ∣ x_{1}, \dots, x_{t - 1})

By minimizing this loss over 15 trillion tokens, the model compresses the statistical regularities of human syntax, code patterns, common sense, and semantic associations into its weight matrices. When you give an LLM three few-shot examples and ask it to complete the fourth, you are watching pure statistical induction at work. It has mapped the empirical distribution of language better than any human ever could.

Deduction: Chain-of-Thought and Verifiable Reasoning

Over the past two years, reinforcement learning with verifiable rewards (RLVR) and test-time search (DeepSeek-R1, OpenAI o1/o3, AlphaProof) unlocked deep deductive mastery.

When you place a reasoning model inside a formal axiomatic system—such as Python syntax, Lean 4 type theory, or the rules of chess—the model does not need to invent new rules. The rules (the axioms) are fixed. The model uses Monte Carlo tree search, beam search, and policy gradients to explore branching deductive chains:

Axioms A \cup Premises P ⊢_{Lean 4} Theorem T

In formal theorem proving, DeepMind’s AlphaProof achieved silver-medal level performance at the International Mathematical Olympiad by chaining thousands of deductive tactic steps in Lean. A model can search millions of deductive paths per second without fatigue. Within a closed system with verified rules, AI is superhuman.

3. What Abduction Truly Is: The Leap Beyond the Box

If induction finds patterns within existing data, and deduction computes consequences within existing axioms, what is abduction?

Abduction is the invention of a new conceptual ontology when the existing rulebook cannot explain reality.

It is often called the “Aha!” moment or insight reasoning. Crucially, abductive leaps possess three characteristics that make them mathematically hostile to standard gradient descent and next-token prediction:

Non-Intuitive or Counter-Intuitive Connections: Abduction does not follow the path of highest probability. It frequently connects two domains that appear completely unrelated or contradictory under the current paradigm.
Inventing New Axioms, Not Solving Within Them: It does not adjust the weights of existing variables; it defines an entirely new variable or throws out a foundational assumption that everyone assumed was sacred.
Born from Sparse or Zero Evidence: An abductive leap usually occurs before the validating data exists. It is not an empirical summary of thousands of data points; it is a creative hypothesis designed to explain a singular, nagging anomaly.

Induction fits rules to observed data and deduction derives consequences of fixed axioms, both inside the current paradigm. Abduction answers an anomaly with a new axiom, which starts a new box.

The Paradigm Wall: In the words of philosopher of science Thomas Kuhn, language models excel at “normal science”—solving puzzles within an accepted paradigm using deduction and induction. Abduction is the spark of “scientific revolutions”—shattering the paradigm and replacing its foundations.

4. Historic Breakthroughs: Connecting the Counter-Intuitive Dots

To see why statistical transformers cannot “jump,” look at how humanity’s greatest intellectual leaps actually occurred:

Kepler, Semmelweis and Einstein each proposed the new idea before the evidence or theory that confirmed it: Rudolphine Tables and the 1631 Mercury transit, Pasteur and Koch, the 1919 eclipse.

Case 1: Albert Einstein and the Falling Elevator (1907)

In 1907, Isaac Newton’s laws of gravitation had reigned supreme for more than two centuries. They predicted the orbits of planets, the tides of oceans, and the trajectory of cannonballs with staggering precision. There was no raging experimental crisis. The slight discrepancy in the perihelion precession of Mercury (a microscopic 43 arcseconds per century) was considered a minor observational nuisance to be absorbed by an undiscovered asteroid.

A pattern-matching language model in 1907 trained on the entire corpus of physics would have assigned a 99.999% probability to the proposition: “Gravity is an attractive force proportional to mass and inverse-square distance.”

Then Einstein had what he called “the happiest thought of my life” (glücklichste Gedanke meines Lebens): “If a person falls freely, he will not feel his own weight.”

Through an embodied, physical thought experiment (a Gedankenexperiment of an elevator falling in a vacuum), Einstein made a wildly counter-intuitive abductive leap: gravity is not an attractive force at all. It is the geometric curvature of four-dimensional spacetime caused by mass and energy.

Notice: the empirical data required to validate General Relativity did not exist when Einstein formulated it. Arthur Eddington did not photograph the bending of starlight around the sun during a solar eclipse until 1919—twelve years later. Einstein didn’t perform induction on existing data. He invented a new axiom that told scientists what data to look for.

Case 2: Johannes Kepler and the Heresy of the Ellipse (1609)

For two thousand years—from Aristotle and Ptolemy through Nicolaus Copernicus and Tycho Brahe—astronomy rested on an unshakeable aesthetic and theological axiom: celestial bodies must move in perfect circles at uniform speeds.

When Copernicus placed the Sun at the center of the solar system, he did not abandon circles; in fact, he had to introduce more complex mathematical epicycles (circles rolling upon circles) than Ptolemy just to make circular orbits fit the sky.

When Johannes Kepler inherited Tycho Brahe’s peerless observational dataset on the orbit of Mars, he spent years trying to fit the data into combinations of circular epicycles. He managed to get the model to agree with observations across almost the entire orbit—except for a tiny discrepancy of 8 arcminutes (less than one-third the diameter of the full moon).

An optimizer would have treated 8 arcminutes as sensor noise or added another small epicycle. Instead, Kepler made the jarring abductive leap: nature does not care about our aesthetic preference for circles. Planets move in ellipses, with the Sun sitting at one focal point.

To 17th-century astronomers, an ellipse was an ugly, asymmetric, defective circle. It was completely counter-intuitive. Yet by shattering the circular axiom, Kepler collapsed decades of complex epicyclic math into three simple, elegant laws.

Case 3: Ignaz Semmelweis and the Invisible Poison (1847)

In the 1840s at the Vienna General Hospital, young mothers were dying of puerperal (childbed) fever at an apocalyptic rate of 10% to 18% in the First Obstetric Clinic (run by doctors and medical students), compared to only 2% in the Second Clinic (run by midwives).

The medical establishment dogmatically blamed the discrepancy on “atmospheric-cosmic-telluric influences” or “miasma” (bad air). When doctors tried to explain why the midwives’ ward had lower mortality, they suggested that the midwife clinic was less crowded, or that mothers in the first clinic suffered psychological trauma from hearing the priest’s bell during last rites.

Then, Semmelweis’s close friend, pathologist Jakob Kolletschka, died of identical symptoms after accidentally being pricked by a student’s scalpel during a post-mortem autopsy.

Semmelweis connected two completely disparate dots: medical students were walking directly from the autopsy room (dissecting rotting cadavers) into the maternity ward to examine laboring mothers without washing their hands. He posited an unheard-of hypothesis: invisible “cadaveric particles” were being transferred from dead bodies into living patients.

He ordered doctors to wash their hands with chlorinated lime before examinations. Maternal mortality plummeted from 18% to under 1.2% almost overnight. Yet because bacteria had not yet been formally identified under microscopes, the medical establishment mocked and ostracized him. Semmelweis had invented the germ theory of disease decades before Louis Pasteur and Robert Koch.

5. The Computational Ceiling: Why Scaling Won’t Produce Abduction

Why can’t we simply train a 10-trillion parameter model with reinforcement learning until it starts discovering new physics?

The answer lies in the mathematical constraints of the optimization loop itself:

1. The Verifier Paradox

In RLVR (reinforcement learning with verifiable rewards), a policy improves because an external verifier grades its outputs. But a verifier can only grade what its author already understands how to check.

If you set up an RL training gym for astronomy in the year 1600, your verifier code would calculate rewards based on how well trajectories aligned with Ptolemaic epicycles. When Kepler’s policy proposed an ellipse, the verifier would flag it as a violation of circular physics and assign it a reward of exactly 0! Reinforcement learning ruthlessly penalizes the very deviations that constitute scientific revolutions.

A verifier written in 1600 would reward one more epicycle and give Kepler's ellipse reward 0: RL can only reward what its verifier already knows how to check.

2. The Closed Symbolic Hull

An LLM operates within a discrete vocabulary embedding space (V ≈ 150,000). Every sentence it generates is an interpolation across the convex hull of human language. It is fundamentally a closed formal system.

As Tom Zahavy points out in LLMs Can’t Jump, human abductive leaps are deeply grounded in embodied physical sensation. Einstein didn’t manipulate symbols in an algebraic solver; he imagined the visceral sensation of falling in an elevator. Kekulé didn’t run an exhaustive combinatorial graph search over carbon bonds; he had a reverie of an ouroboros—a snake biting its own tail—and made the leap to the benzene ring.

Without physical grounding in a multi-modal world model governed by the invariant laws of matter and energy, pure statistical language models remain trapped in the “Chinese Room” of symbolic recombination.

6. The Division of Labor: What AI Does Best vs. What Humans Must Do

Recognizing the limits of AI is not pessimism—it is the hallmark of serious engineering.

We do not need language models to be Albert Einstein to unlock trillions of dollars in economic value. In modern civilization, 99.9% of real-world productivity is induction and deduction:

Synthesizing 50,000 lines of bug-free Verilog RTL from an architectural specification (Deduction).
Mining petabytes of genomics data for candidate oncology biomarkers (Induction).
Auditing multi-hundred-page commercial contracts against regulatory statutes (Deduction).
Compiling, testing, and optimizing high-throughput GPU kernels (Deduction & Empirical Induction).

These are the exact enterprise domains where platforms like g factor operate: giving organizations the power to post-train open models on their own proprietary workflows, equipping them with deterministic verification gyms (RLVR), and serving them with zero data leakage.

The division of labor for the next century of science and industry is crystal clear:

The Human Role: Identify the anomaly. Notice the 8-arcminute discrepancy that everyone else ignored. Wonder why doctors’ patients die while midwives’ patients live. Formulate the bold, counter-intuitive hypothesis. Make the abductive jump.
The AI Role: Take the new axiom and unpack its consequences across a million permutations. Verify the theorems, run the numerical simulations, test every corner case, and write the code. Execute the deductive and inductive marathon with superhuman speed.

The division of labour the article proposes: people notice the anomaly and propose the new axiom; models derive its consequences and test them at scale.

AI will not replace the spark of human genius. It will give that spark a multi-million-node engine to reshape reality.

Applying this to your own models? We build private RL training environments with deterministic verifiers for teams' own workflows. Private training environments.

Originally published at g-ftech.com.

Physical AI: Why the Next Big Frontier Is Giving Software Agents Hands

Aleksei Romanov — Wed, 30 Sep 2026 04:22:58 +0000

For the last three years, the tech industry has treated artificial intelligence as a ghost trapped behind a glass screen: an entity that answers emails, writes code, summarizes meetings, and argues on the internet.

We have become so accustomed to treating Large Language Models as purely digital conversationalists that we missed the profound architectural truth hiding in plain sight: an AI that can write, debug, and execute complex code across a 40-step autonomous loop has already solved the hardest half of robotics.

Look at what modern coding and research agents do every day. The model inspects a codebase, breaks down a long-horizon goal into sub-tasks, calls an API tool, observes the terminal output or compiler error, reflects on what went wrong, adjusts its mental model, and executes the next tool call. That is not simple text prediction. That is embodied cognition—it just happens to be operating inside an operating system instead of a physical workshop.

This is why Physical AI is the next massive frontier. To a reasoning model, the real world—whether it is a 6-axis robotic arm, an automated chemistry liquid handler, a 5-axis CNC milling machine, or a bipedal humanoid—is not an alien realm. To a reasoning agent, physical hardware is just another API endpoint.

Below, we break down why the leap from software agents to physical machines is happening now, explore Anthropic’s groundbreaking Model Hardware Standard (MHS), examine OpenAI’s relaunch of in-house humanoid robotics, walk through a case study of contact-rich gear assembly in Isaac Lab and on real robots, and explain why training models inside verifiable physical gyms (RLVR) is the only way to build embodied intelligence without smashing million-dollar hardware.

1. The Proven Loop: Long-Horizon Tool Use Is Already Solved

Classical robotics research spent four decades bogged down in an end-to-end purist trap: the belief that to make a robot smart, you had to train a single monolithic neural network that takes raw camera pixels and directly outputs raw motor voltages or joint torques at 1,000 Hz.

That approach failed to scale because it tangled two completely different problems:

High-level reasoning: Understanding the world, planning long-horizon strategies, and adapting to failures.
Low-level motor control: Keeping balance, calculating inverse kinematics, and dampening joint oscillations.

Human beings don’t work that way either. When you brew a cup of coffee, your prefrontal cortex doesn’t consciously calculate the voltage curves of your forearm bicep fibers. Your conscious brain issues high-level semantic directives: “Pick up kettle → check water level → pour 200 ml over grounds → inspect extraction color.” Your cerebellum and spinal cord handle the millisecond-level motor physics underneath.

Split the problem: an LLM agent plans in seconds and chooses tool calls; a low-level controller tracks trajectories and joint torques at 100–1,000 Hz.

Over the past two years, frontier language models have undeniably mastered the “prefrontal cortex” role:

They can plan 30-to-50 step dependency graphs without losing context.
They make structured tool calls with typed arguments (JSON / schemas).
When an action returns an unexpected error code or unexpected sensor state, they don’t crash—they debug and course-correct.

Whether that tool call is git_commit(), sql_query(), or robotic_arm_grip(force=12N) makes virtually zero mathematical difference to the model’s internal reasoning loop.

2. The Physical Bridge: Hardware as a Stateful API

Walk into a modern advanced chemistry laboratory, an automated semiconductor cleanroom, or a high-precision machining shop. What do you see?

You do not see analog dials and pulleys. You see automated instruments controlled by embedded microcontrollers, exposing network interfaces:

Automated Wet Labs: Hamilton liquid handlers, Tecan microplate readers, automated centrifuges, and mass spectrometers communicating over REST or gRPC.
Precision Manufacturing: Haas and DMG MORI CNC milling machines accepting ISO 6983 G-code over industrial Ethernet.
Semiconductor & Photonics: Laser calibration stages, optical alignment piezos, and vacuum chamber valves running via Modbus or SCPI protocols.
Robotic Manipulation: Collaborative arms (UR10e, Franka Emika) accepting Cartesian trajectory waypoints.

To an AI agent, an automated chemistry workstation looks identical to a cloud infrastructure platform:

Agentic Concept	Digital Software Agent (e.g. Cursor / Devin)	Physical AI Agent (e.g. Wet Lab / Robotic Cell)
Environment State	Repository files, git status, database rows.	Plate rack coordinates, reagent fluid volumes, joint encoders, thermal cameras.
Action Primitive (Tool Call)	`write_file(path, diff)`	`aspirate_reagent(channel=4, volume_uL=50)`
Sensory Feedback	Compiler stderr, unit test assertion logs, terminal exit code.	Spectroscopy UV absorption curves, pressure sensor deltas, computer vision bounding boxes.
Course Correction	Fixes syntax error on line 42 and re-runs test suite.	Detects liquid foaming, lowers pipette aspiration velocity, and re-draws sample.

The cognitive loop is 100% identical: Reason → Act via Tool → Observe Physical Feedback → Reason Again.

The same reason → act → observe → correct loop drives a coding agent and a lab robot; only the tools and sensors differ.

3. Contact-Rich Reality: Isaac Lab Gear Assembly Walkthrough

To see how this works on real hardware, consider an industrial benchmark: gear assembly.

A robot holds a precision gear above an unfinished mechanism. Its task: line up the central bore with a shaft, lower the gear until its teeth mesh smoothly with a neighboring gear, seat it firmly, and release.

Gear assembly is a classic contact-rich task: touching the parts physically alters what the robot must do next. A tiny 0.5 mm positioning error can leave the gear rim resting on top of the shaft instead of sliding down. In software, if an API call fails, repeating the same request might eventually work. In the physical world, repeating the same downward push will bend the robot’s gripper, gouge the brass shaft, or trigger a motor torque fault.

NVIDIA’s Isaac Lab gear-assembly tutorial shows a control policy doing this both in GPU simulation and on a real Universal Robots UR10e arm. The figure below breaks the task into its six-stage closed loop, including what the policy does when the gear is misaligned:

Gear assembly as a six-step closed loop. A force spike at contact triggers retract, re-estimate and re-approach, and success is checked after the gripper releases.

The original post plays NVIDIA’s sim-to-real clip next to an interactive 3D walkthrough of this loop.

Deconstructing the Hierarchical Control Layers

Notice how the gear assembly workflow cleanly separates concerns across architectural layers:

Layer	Gear Assembly Implementation	What It Requires to Succeed
High-Level Task Planner	Orchestrates the macro sequence: `observe() → approach() → try_insert() → seat() → release() → verify()`.	Long-horizon task goal, state machine recovery transitions, and high-level tool definitions.
Perception Engine	Estimates the gear bore center, shaft axis, and neighboring gear tooth angles from depth cameras.	Calibrated transforms, sub-millimeter pose estimation, and uncertainty estimation.
Motion & Compliance Controller	Tracks the trajectory while monitoring 6-axis force-torque sensors to detect unexpected resistance.	Real-time 100–500 Hz control loop, impedance control, joint torque limits, and collision avoidance.
Independent Result Verifier	Checks whether the gear remains seated at target depth after the gripper releases and pulls away.	Independent physical ground truth, not just an API return code of “success.”

In software, an API call returning status code 200 OK usually means the job is done. In robotics, “the move command completed” is not proof that the assembly succeeded. The part could have jammed, slipped out of the gripper fingers, or bounced. Real physical intelligence requires closing the loop: verify the physical state before declaring victory.

4. Standards Are Arriving: From MCP to Anthropic’s MHS

In software, agentic tool use exploded because the industry agreed on a standard. In late 2024, Anthropic introduced the Model Context Protocol (MCP) (and later donated it to the Linux Foundation’s Agentic AI Foundation alongside OpenAI and Block). MCP solved the N × M integration problem: instead of writing custom code to connect every model to GitHub, PostgreSQL, Slack, and Google Drive, developers wrote a single MCP server.

In August 2026, Anthropic took the inevitable next step by previewing the Model Hardware Standard (MHS).

What is the Model Hardware Standard (MHS)? Developed in collaboration with premier research institutions like the Howard Hughes Medical Institute (HHMI) Janelia Research Campus, MHS is an open, model-agnostic specification that allows AI agents to safely discover, calibrate, and operate physical lab and factory equipment. Just as MCP gives models a universal socket to read software files, MHS gives models a universal socket to command microscopes, robotic arms, microfluidic pumps, and optical alignment stages.

A shared protocol replaces one custom integration per model–instrument pair (N × M) with one adapter per side (N + M): MCP for software tools, MHS for instruments.

Early field reports with MHS are remarkable:

Parallel Scientific Workflows: Agents can orchestrate multiple disparate instruments in parallel—running a multi-day drug discovery assay by scheduling liquid handlers, centrifuges, and fluorescence microscopes in synchrony.
Understanding Physical Failure Modes: In tests with Genentech, when an automated protein purification run hit an error due to liquid foaming, the agent recognized that the failure was not a software bug requiring a code retry, but a physical surface-tension breakdown requiring a lower pumping speed.
Universal Hardware Drivers: Hugging Face is adding MHS support into their robotics library (LeRobot), and hardware teams from Raspberry Pi to industrial automation vendors are shipping native MHS drivers.

Once hardware has a universal USB-like driver standard that models can query, the barriers separating software intelligence from physical execution dissolve.

5. The Big Lab Rush: OpenAI, Humanoids, and Hardware-Software Co-Design

If anyone doubted where the frontier of AI is heading, OpenAI answered the question unequivocally in mid-2026.

After quietly shutting down its original robotics team in 2021 to focus on pure language models, OpenAI officially relaunched its Robotics Division under VP of Research Aditya Ramesh. And on the Sources podcast, CEO Sam Altman confirmed the ambition that had been whispered about for months: “We will definitely do a humanoid robot. We will do other form factors as well.”

Why Are the Frontier Labs Rushing into Hardware?

Two massive economic and technical realities are driving this shift:

The Digital Data Wall: Frontier models have already ingested virtually the entire public internet. You cannot 10x pre-training data by scraping Reddit again. The physical world—how materials deform, how liquids pour, how tools interact with surfaces, how objects balance—is an infinite, untamed ocean of high-bandwidth training ground truth.
The $5 Trillion Physical Economy: Software is huge, but the physical economy—construction, semiconductor fabrication, precision machining, agriculture, pharmaceutical research, and logistics—dwarfs it. Building models that can only manipulate pixels captures a fraction of global GDP; building models that can safely manipulate matter captures the rest.

Furthermore, the leading players are converging on Hardware-Software Co-Design. Instead of buying off-the-shelf industrial robots designed in 1995 for rigid factory loops and trying to bolt modern neural nets onto them, teams are designing actuators, compliance sensors, and joint geometries specifically tailored to what high-level neural policies need.

6. The Training Requirement: Why Physical AI Demands Verifiable Gyms (RLVR)

Here is the fundamental catch of Physical AI: you cannot train a reinforcement learning agent by trial-and-error on a real $500,000 physical machine.

In software, if a model hallucinates a bad pointer or writes an infinite loop, you kill the process, give it a reward of 0, and restart. In the physical world, if an untrained 27B policy outputs a random trajectory command, it snaps a diamond-tipped CNC cutter, crashes a pipette head through a quartz plate, or sends a 150-pound humanoid crashing into a wall.

This is why Physical AI requires Reinforcement Learning with Verifiable Rewards (RLVR) inside high-fidelity task gyms before a single motor moves in the real world:

Kinematic & Physics Simulation (Isaac Lab, MuJoCo): As in the gear assembly example above, the agent practices grasping, aligning, and inserting objects millions of times across parallel GPU instances. Using domain randomization (systematically varying friction coefficients, part clearances, masses, and lighting during training), the policy develops robust compliance that transfers seamlessly across the “sim-to-real” gap.
Computational Fluid & Thermal Gyms (OpenFOAM): For aerodynamic shaping or chemical reactor control, aerodynamic profiles and nozzle designs are evaluated against numerical convergence and Navier-Stokes solver checks.
Parametric CAD & CAM Validation (CadQuery, G-Code Simulators): The model learns to generate machinable mechanical components verified against collision-safe 3-axis toolpaths and geometric tolerances.
Formal Circuit Verification (Verilog RTL / SPICE): Designing the custom silicon and motor controllers that power next-generation robots, verified against cycle-accurate timing simulation.

Train in a verifiable simulated gym with domain randomization, gate the move to real hardware, and feed real-world failures back into simulation.

By proving its reasoning inside deterministic digital gyms first, the agent arrives at the physical machine not as a reckless novice, but as a qualified, safety-verified operator.

7. The Physical World Is the Ultimate Gym

The history of modern AI has been a steady march across expanding arenas.

First, AI conquered closed, discrete games: chess, Go, StarCraft. Next, it conquered digital language: conversation, essay writing, translation. Then, it conquered software engineering: codebases, syntax trees, compilers, and APIs.

The final, greatest arena was always going to be the physical world. Molecules, metal, circuits, glass, and steel.

The models are ready. The long-horizon reasoning loops work. The tool-calling protocols (MCP and MHS) are standardizing. And the world’s most ambitious engineering labs are building the bodies.

Don’t keep intelligence locked behind glass. Give it hands.

Applying this to your own models? We build private RL training environments with deterministic verifiers for teams' own workflows. Private training environments.

Originally published at g-ftech.com.

Distributed Training & Inference: From CPUs and GPUs to a Cluster

Aleksei Romanov — Tue, 29 Sep 2026 19:36:30 +0000

You can run a small model on a laptop, train a larger one on a GPU server, and spread an enormous one across a cluster. The difficult step is understanding what changes between those setups. Adding GPUs gives you more arithmetic capacity and more memory, but your program must decide how to use both—and how to pay for moving data between them.

Start with one question: what are we dividing? Different training examples, different layers, pieces of one matrix, or the stored state used to update the model? Those choices explain data parallelism, pipeline parallelism, tensor parallelism, and sharding. CUDA, NCCL, FSDP, DeepSpeed, and Ray then have distinct jobs within the resulting system.

Start here

Start with one machine, then choose what to divide: examples, layers, matrix operations, or stored training state. Each illustration changes one of those choices.

Training / inference: Updating a model using data / running the trained model to get an output.

Weights: The model’s learned numbers. A matrix is a rectangular table of such numbers; a tensor generalizes an array to more dimensions.

Gradient: The loss function’s sensitivity to a parameter, used by an optimizer to decide an update.

Node / worker: One server / one process doing part of the job.

1. CPU, GPU, and the role of CUDA

A CPU is a flexible processor. It runs the operating system, handles application logic, prepares data, and performs numerical computation. Modern CPUs have multiple cores and vector instructions, so “CPU” does not mean “one calculation at a time.” They are particularly useful when work involves varied instructions, branches, or a modest amount of arithmetic.

A GPU devotes more of its design to doing similar arithmetic across many values. A neural network repeatedly multiplies matrices: the same pattern of multiplication and addition applies to many rows, columns, and examples. That gives the GPU a large amount of parallel work. Small operations, irregular access, or frequent transfers can leave it underused; a GPU does not accelerate arbitrary code simply because it is installed.

In a typical discrete-GPU server, the CPU uses host RAM and each GPU has its own device memory, often called VRAM. The CPU launches a kernel—a function executed on the GPU by many threads. Data must be accessible to that device; weights and intermediate results can remain there across many operations. Some systems instead use unified memory architectures, so separate physical memory pools are not universal. NVIDIA’s programming model.

CUDA is NVIDIA’s parallel-computing platform and programming stack. It provides the runtime, programming tools, and libraries through which frameworks can execute NVIDIA GPU operations. In a usual PyTorch CUDA setup, your Python matrix multiplication dispatches to GPU kernels or libraries such as cuBLAS; you do not need to write a CUDA kernel yourself. A compatible NVIDIA driver and CUDA-enabled framework are needed for that path. Installing a compiler toolkit separately is not always necessary when the framework package already supplies its runtime dependencies. The CUDA platform.

CUDA is not a requirement for every GPU. It is the stack used by the NVIDIA path described here. AMD has ROCm, and PyTorch can use Apple GPUs through its MPS backend. A GPU needs a supported software backend; that backend need not be CUDA.

A CPU runs varied work on a few flexible cores; a GPU runs the same arithmetic on many values. One line of PyTorch reaches the GPU through the CUDA runtime and libraries such as cuBLAS.

The original post has interactive 3D versions of these diagrams.

2. One GPU, several GPUs, several nodes

A node is one server: CPUs, RAM, network interfaces, and possibly several GPUs. A single-node, multi-GPU job uses multiple devices inside that server. A multi-node, multi-GPU job crosses server boundaries as well. A common arrangement runs one training process per GPU. Each process receives a rank, its identifier within the distributed job; the total process count is the world size.

Inside a node, devices communicate over PCIe and, on supported systems, faster GPU fabrics such as NVLink/NVSwitch. Between nodes, traffic travels through network adapters and a network such as Ethernet or InfiniBand. The relevant questions are bandwidth, latency, contention, and actual connectivity—not just the number of GPUs. The fabric names alone do not establish the speed of a particular deployment.

Four 24 GB GPUs do not automatically behave like one 96 GB GPU. If your program creates a complete 40 GB model allocation on every device, it still fails. You need a method that explicitly partitions the model or its state. Fast links help move those partitions; they do not remove the need to partition them.

From one GPU to a cluster. Inside a node GPUs talk over NVLink or PCIe; across nodes the link becomes the network. The hybrid layout is 2 data-parallel replicas × 2 pipeline stages × 2 tensor shards.

The hybrid view uses eight GPUs: two tensor shards per stage, two pipeline stages per model replica, and two data-parallel replicas. Each node contains one complete replica spread across four GPUs. This placement keeps the frequent tensor exchanges local and synchronizes corresponding training shards across nodes. Other mappings are possible; choose one for the actual communication pattern and hardware.

N_{GPU} = D \times T \times P = 2 \times 2 \times 2 = 8

Here D is the data-parallel degree, T the tensor-parallel degree, and P the pipeline degree. This is the illustrated three-axis layout, not a universal formula covering every expert-, context-, or hybrid-sharding arrangement.

3. Data parallelism: same model, different examples

Imagine four people solving different pages of practice problems from the same textbook. Each has a complete copy of the model. Each processes different examples and computes gradients: the suggested changes to its parameters. Before the optimizer updates those parameters, the workers combine their suggestions.

Green is the part of the model a GPU holds. Data parallelism duplicates the model and splits the batch; tensor parallelism cuts inside layers and needs NVLink-class bandwidth; pipeline parallelism cuts between layers and pays in idle time at the edges of every batch.

In PyTorch DistributedDataParallel (DDP), gradient communication uses all-reduce and can overlap with backpropagation. With equally sized local batches and mean losses, averaging the local gradients gives the mean gradient over their combined batch. Matching initial weights and optimizer states then let each worker take the same update. DDP does not divide your dataset automatically; the input pipeline must give workers the intended distinct samples. PyTorch DDP tutorial.

Data parallelism: each GPU holds the full model and computes a gradient on its own mini-batch (1, 3, 5, 7); all-reduce averages them to 4 so every GPU takes the same step.

B_{effective} = b_{micro} \times D \times A

Here b is the number of examples per microbatch on each data-parallel replica, D is the number of those replicas, and A is the number of microbatches accumulated before an optimizer update. With b = 2, D = 4, and A = 8, the effective batch is 64 examples. Tensor and pipeline workers cooperate on the same examples, so their counts do not multiply this batch size. For variable-length language-model data, also track tokens and the loss-normalization convention.

Data parallelism is useful when the model and training state already fit and you need to process more data per second. Replicated DDP keeps a full model and optimizer state on each GPU; it does not solve their memory footprint. Doubling the global batch can also change optimization behavior, so compare time to a target quality as well as raw step time.

4. Tensor parallelism: divide the work inside a layer

“Weight parallelism” is ambiguous. It can mean splitting a matrix calculation across devices, which is tensor parallelism (TP), or storing parameter shards between calculations, which we discuss under FSDP. The distinction is what each GPU computes after the split.

Take a linear layer y = Wx, with x represented as a column vector. If GPU 0 gets the top rows of W and GPU 1 gets the bottom rows, both read x and produce different coordinates of y. Joining those output slices recovers the complete result:

W = [W_{0} W_{1}], y = [W_{0} x W_{1} x]

Alternatively, split W into left and right column blocks, and split x accordingly. Now both devices contribute to every output coordinate. Their results must be added:

W = [W_{0} W_{1}], x = [x_{0} x_{1}], y = W_{0} x_{0} + W_{1} x_{1}

Tensor parallelism on y = Wx: split W by rows and concatenate the output slices, or split W and x by columns and add the partial results.

Joining slices may use all-gather; summing contributions may use all-reduce or reduce-scatter. An optimized model can keep intermediate outputs sharded, so it does not necessarily gather after every layer. Library names such as “row-wise” and “column-wise” also depend on weight-storage conventions; the formulas above define the axes used in this example.

TP spreads both a layer’s parameters and its arithmetic across GPUs. It can make a large layer fit, but requires repeated communication through the network of layers. That is why fast connections matter so much, particularly when generating one token at a time. PyTorch tensor parallelism.

5. Pipeline parallelism: divide the sequence of layers

Pipeline parallelism (PP) assigns successive groups of layers to different devices. GPU 0 computes early layers, passes their activations to GPU 1, and so on. Backpropagation sends activation gradients in the reverse direction. Each stage owns its assigned layers rather than a complete model.

A single microbatch leaves most stages waiting. Split a batch into multiple microbatches and a stage can start the next one while its neighbor works on the previous one—like different stations on an assembly line. Idle periods while the pipeline fills or drains are called bubbles.

Pipeline bubbles in a forward-only schedule with 4 equal stages: 1 microbatch keeps 25% of GPU time busy, 4 microbatches about 57%.

U_{forward} = \frac{MP}{P ( M + P - 1 )} = \frac{M}{M + P - 1}

In this deliberately simple forward-only schedule, P equal-duration stages process M microbatches in M + P − 1 ticks. Four stages and one microbatch occupy 25% of the stage-time slots; four microbatches occupy 4/7, or about 57.1%. This compares two schedules with different amounts of work—it is not a measured speedup.

Training adds backward passes and saved activations. Schedules such as GPipe and one-forward-one-backward (1F1B) organize those passes differently. Uneven stage workloads, communication, and activation storage change real utilization. More microbatches can reduce bubbles, but tiny microbatches may use each GPU poorly. PyTorch pipeline schedules.

6. FSDP and DeepSpeed: stop duplicating training state

Training stores more than the weights. It also needs gradients, optimizer state such as Adam’s running moments, activations for backward, and temporary buffers. Sharding means each worker owns only a portion of some of that state.

DeepSpeed is a training and inference optimization library. ZeRO, its Zero Redundancy Optimizer approach, progressively removes duplicated training state from data-parallel workers. The stage number describes what is partitioned; it is not a ranking in which the highest number always runs fastest. DeepSpeed’s ZeRO guide.

Method	Weights	Gradients	Adam moments	Subtotal
DDP	2 GiB	2 GiB	8 GiB	12 GiB
ZeRO-1	2 GiB	2 GiB	2 GiB	6 GiB
ZeRO-2	2 GiB	0.5 GiB	2 GiB	4.5 GiB
ZeRO-3 / full sharding	0.5 GiB	0.5 GiB	2 GiB	3 GiB

These illustrative sizes correspond to BF16 weights and gradients plus two FP32 Adam moments, with no separate FP32 master weights. If your implementation maintains a master copy, include it in the optimizer-related state before dividing that state across workers. Actual dtypes and allocations depend on configuration.

Per-GPU persistent training state with 4 workers for the toy model: DDP 12 GiB, ZeRO-1 6 GiB, ZeRO-2 4.5 GiB, ZeRO-3 / FSDP 3 GiB. Peak memory is higher.

Fully Sharded Data Parallel (FSDP) is PyTorch’s mechanism for sharding parameters, gradients, and optimizer state. In a common fully sharded configuration, workers gather the weights for the next wrapped unit, compute it on their different local examples, and release the unsharded weights afterward. Backward computation gathers what it needs again; reduce-scatter combines gradients and leaves each worker with its owned portion for the optimizer update. PyTorch FSDP2 tutorial.

That is the key difference from TP: FSDP gathers a unit so each data-parallel worker can compute that unit on its own examples; TP divides the unit’s computation across workers. Fully sharded FSDP and ZeRO-3 share the broad memory-saving idea, but their implementations, interfaces, and available policies are not identical.

M_{peak} \approx M_{persistent shards} + M_{live gathered weights} + M_{activations} + M_{temporary}

A 3 GiB state subtotal does not promise a 3 GiB peak. Wrapping an entire model as one gather unit can create a much larger temporary allocation than gathering layer by layer. Prefetching may keep additional units live. Activation checkpointing trades recomputation for less saved activation memory; CPU or NVMe offload trades device memory for transfers and potentially slower steps. These are complementary choices, not extra ZeRO stages.

7. Ray starts work; NCCL exchanges GPU tensors

Ray is a distributed-computing framework. A Ray task is a function scheduled for remote execution; an actor is a stateful worker whose methods can be called remotely. This is useful when a workload combines GPU learning, CPU data preparation, environment simulations, and services with different resource needs. Ray Core concepts.

Ray Train packages the training function, worker count, and resource requirements into a distributed job. It starts workers and sets up the framework’s distributed environment. Your training code still chooses DDP, FSDP, or another supported strategy. Requesting four GPUs does not automatically split an arbitrary model across them. Ray Train overview.

NCCL, the NVIDIA Collective Communications Library, handles common GPU communication operations. All-reduce combines contributions and gives every participant the result. All-gather assembles everyone’s slices on each participant. Reduce-scatter combines contributions but leaves each participant with only its result slice. DDP and sharded training use these building blocks differently. NCCL collective operations.

torchrun and Ray Train differ only in how workers are started; PyTorch still chooses DDP or FSDP, CUDA runs local kernels, and NCCL moves tensors between GPUs.

Ray is optional for cluster training. PyTorch’s torchrun can launch distributed workers across nodes, often within resources allocated by Slurm or Kubernetes. Ray adds orchestration capabilities; it is not a mathematical requirement for distributed learning and does not replace CUDA or NCCL.

Think in layers: the cluster manager allocates machines; a launcher or Ray starts workers; PyTorch and the chosen strategy define their computation; CUDA executes local NVIDIA GPU work; NCCL exchanges tensors between those workers. A Ray dashboard showing four available GPUs proves resource discovery. Successful, synchronized optimizer steps are separate evidence that training works.

8. Inference changes the objective

Ordinary inference uses fixed weights to produce outputs. It has no training backward pass, gradients, or optimizer updates. Autoregressive language-model serving instead stores a KV cache: attention keys and values from previous tokens, reused when generating the next token.

During prefill, the model processes the prompt, providing substantial parallel work across prompt tokens. During decode, each sequence advances one generated token at a time. Batching multiple sequences improves the amount of useful work per step, while longer contexts and more concurrent requests increase cache pressure. Request latency, total tokens per second, and maximum concurrency are different goals.

Training keeps gradients and optimizer state and all-reduces every step; serving replicas keep a KV cache and never synchronize; one sharded model exchanges activations every layer.

If a model fits comfortably on one device, independent replicas can serve different requests. They do not average gradients. If one model replica needs multiple GPUs, TP and PP divide it so those GPUs cooperate on requests. You can then replicate that entire group for more serving capacity. Splitting onto more devices can increase communication enough to hurt latency even when memory usage per device falls. vLLM parallelism and scaling.

FSDP’s repeated gather-and-release pattern is designed around training-state savings; it is usually not the default layout for low-latency token serving. An inference engine has its own weight placement, kernel, batching, and KV-cache decisions. Separating prefill and decode onto different workers is another serving option, but transferring the cache introduces an additional cost.

9. Choose the split that addresses the bottleneck

Observed problem	Candidate	Cost to check
Training fits, but needs more compute throughput	DDP, after profiling the input pipeline	Gradient communication and changed global batch
Optimizer and gradient state exceed VRAM	ZeRO-1/2 or full sharding	State dtypes, communication, peak live allocations
The complete weights do not fit	FSDP / ZeRO-3, TP, PP, or a combination	Largest gathered unit and per-layer working memory
One layer is too large to compute locally	TP, possibly combined with sharding	Communication frequency and fast-link placement
Many layers fit separately across devices	PP	Stage balance, microbatch efficiency, bubbles
A fitting model needs more serving throughput	Batching and independent replicas	Tail latency, cache capacity, request mix
CPU environments and GPU learners need coordination	Ray tasks/actors and training workers	Queueing, placement, failure recovery, stale results

Specialized models add other axes. Expert parallelism places different mixture-of-experts networks on different devices and routes tokens to their selected experts. Sequence parallelism partitions selected activation operations along the sequence dimension; context parallelism distributes long-context attention work and exchanges the information that attention still needs. These are additional tools for particular model structures and activation bottlenecks, rather than synonyms for data parallelism. Megatron Core parallelism guide.

For a useful comparison, hold the model, precision, sequence-length distribution, effective batch, and quality target constant. Separate initialization and compilation from steady-state execution. Record peak memory, useful tokens per second, time waiting for data or collectives, and end-to-end cost. For inference, also record time to first token and latency between generated tokens under the same request load.

The practical progression is to make one device correct, measure what limits it, and introduce the split that removes that limit. A cluster succeeds when workers spend enough time doing useful computation to repay the coordination and communication they introduce.

Applying this to your own models? We post-train open models on teams' own tasks with SFT and GRPO, and hand back the weights. LLM post-training with g factor.

Originally published at g-ftech.com.