You edit one line in config.json — rope_theta: 10000.0 becomes 1000000.0 — restart vLLM with --max-model-len 131072, and your needle-in-a-haystack test at 64k goes green. Then your function-calling eval, which averages 900 tokens per prompt, drops a couple of points. Nothing else changed. No weights moved. No sampling params changed.
That is not noise, and it is not a bug in your serving stack. RoPE scaling buys long-range positional coverage by spending short-range positional resolution, and the exchange rate is computable from the config you just edited.
TL;DR
- RoPE gives dimension pair
iof each head a wavelengthλ_i = 2π · base^(2i/d). Raisingrope_thetastretches every wavelength excepti=0, and stretches the high-index dims the most. - Stretching wavelengths moves whole bands of dimensions out of the range where they can discriminate nearby positions. With
d=128, going from base 1e4 to 1e6 drops the number of dimension pairs with sub-2048-token wavelength from 40 to 27 — a third of your short-range positional bandwidth, gone. - Linear position interpolation (dividing positions by
s) is worse for short context: it compresses the high-frequency dims that encode adjacency. - YaRN (NTK-by-parts) fixes this by only interpolating the dims that never completed enough rotations during training, leaving the fast dims alone, plus an attention temperature of
0.1·ln(s) + 1to re-sharpen logits. - Static YaRN applies the scaling to every request, including your 400-token ones. Enable it only when you actually need the length, or use dynamic scaling and accept that it complicates prefix-cache reuse.
What does rope_theta actually control?
Rotary position embedding splits each head's d-dimensional query and key into d/2 2-D pairs and rotates pair i at position m by angle m · θ_i, where θ_i = base^(-2i/d). The dot product between a query at m and a key at n then depends only on m - n. That relative property is why RoPE extrapolates at all.
The useful way to read that formula is as a wavelength per dimension pair: λ_i = 2π / θ_i = 2π · base^(2i/d). With d = 128 and base = 10000:
| dim pair | wavelength (tokens) |
|---|---|
| 0 | 6.3 |
| 16 | 63 |
| 32 | 628 |
| 48 | 6,283 |
| 63 | ~57,000 |
Each head carries a bank of positional "clocks" spanning four orders of magnitude. Fast dims resolve adjacency — which of these two tokens came first. Slow dims resolve document-scale position — beginning versus middle. A dim is only informative over roughly half a wavelength; past that it wraps and starts aliasing.
Why does raising rope_theta break short-context accuracy?
Because base scaling is not uniform, and the dims it hurts are the ones doing mid-range work.
Note that λ_0 = 2π · base^0 = 2π regardless of base. The very fastest dim is untouched. The multiplier on λ_i is (new/old)^(2i/d), so it grows with i. Going from 1e4 to 1e6 at d=128:
- dim 16: 63 → 199 tokens (3.2×)
- dim 32: 628 → 6,283 tokens (10×)
- dim 48: 6,283 → 198,700 tokens (31.6×)
Dim 32 used to be the clock for "roughly where in this 600-token span are we." Now it takes 6,000 tokens to complete a rotation, so inside a 900-token prompt it barely moves. Its contribution to the attention logit is nearly constant across the whole prompt — a bias, not a signal.
Count the dims that still complete a full rotation inside 2048 tokens. Solve 2π · base^(i/64) < 2048:
- base 1e4:
i < 40→ 40 of 64 pairs - base 1e6:
i < 27→ 27 of 64 pairs
You deleted a third of the positional bandwidth available at typical prompt lengths, and the model's attention heads were trained against the old allocation. The heads that learned to read dim 32-40 for paragraph-level ordering now read a nearly-DC signal. That is the mechanism behind the eval drop, and it shows up hardest on tasks with position-sensitive structure: multi-item ordering, "the third tool result", diff-style comparisons, long system prompts with numbered rules.
Fine-tuning after the base change largely repairs this — the model reallocates which dims it trusts. Llama 3.1 shipping with rope_theta = 500000 and Qwen with 1e6 are trained that way, not patched at inference time. What does not work is editing the number on a checkpoint trained at 10000 and expecting short prompts to behave.
Why doesn't linear position interpolation fix it either?
Position interpolation divides positions by the scale factor s before rotation, so a 32k sequence maps into the 8k range the model saw in training. Every angle shrinks by s.
That is strictly worse for short context than base scaling. Base scaling leaves the fast dims alone; PI squeezes them the most in relative terms. Adjacent tokens that were separated by a rotation of 2π/6.3 now differ by 2π/(6.3·s). At s=4 the model must resolve token order from a quarter of the angular separation it trained on. PI works, but only with fine-tuning, and it is why naive PI reports degraded local-detail behavior.
NTK-aware scaling was the first fix: instead of scaling positions, scale the base so that the highest-frequency dims are almost untouched and only the slow dims get effectively interpolated. Better zero-shot, but it is still a blunt instrument — every dim is adjusted by a smooth function of i, including the ones that needed nothing.
What does YaRN do differently?
YaRN makes the interpolate-or-extrapolate decision per dimension, based on how many rotations that dim completed inside the original training context.
Define r_i = L_orig / λ_i, the rotation count. If r_i > β (default 32), the dim has seen many full periods during training; it can extrapolate safely, so leave it alone. If r_i < α (default 1), the dim never completed even one rotation, so every long-context position is genuinely out of distribution; interpolate it fully by s. Between them, ramp linearly.
import math
def yarn_inv_freq(dim=128, base=10000.0, scale=4.0, L_orig=8192,
beta_fast=32, beta_slow=1):
"""Per-dimension-pair inverse frequencies under YaRN (NTK-by-parts)."""
exps = [2 * i / dim for i in range(dim // 2)]
extrap = [base ** -e for e in exps] # untouched
interp = [(base ** -e) / scale for e in exps] # stretched by s
def dim_at_rotations(r):
# index whose wavelength completes exactly r rotations in L_orig
return dim * math.log(L_orig / (r * 2 * math.pi)) / (2 * math.log(base))
low = math.floor(dim_at_rotations(beta_fast)) # fast side
high = math.ceil(dim_at_rotations(beta_slow)) # slow side
out = []
for i, (e, f) in enumerate(zip(extrap, interp)):
ramp = 0.0 if i <= low else 1.0 if i >= high else (i - low) / (high - low)
out.append(e * (1 - ramp) + f * ramp)
return out
# attention temperature: multiply cos/sin tables by this
attn_factor = 0.1 * math.log(4.0) + 1.0 # ~1.139 at s=4
Run the boundaries for d=128, base=1e4, L_orig=8192: low = 25, high = 50. Dims 0-25 are left exactly as trained. Dims 50-63 are fully interpolated. Only the 24 dims in between get a blend. Your adjacency clocks survive untouched — that is the whole trick.
The second half of YaRN is the attention temperature. Longer sequences mean more keys competing in the softmax, which raises attention entropy and flattens the distribution. YaRN counters it by scaling logits, implemented for free by multiplying the cos/sin tables by sqrt(1/t) = 0.1·ln(s) + 1. At s=4 that is a 1.139× sharpening. It costs nothing at runtime and it is the difference between "works" and "works well" in the paper's perplexity numbers.
Config side, HF and vLLM take the same shape:
{
"rope_scaling": {
"rope_type": "yarn",
"factor": 4.0,
"original_max_position_embeddings": 32768,
"beta_fast": 32,
"beta_slow": 1
},
"max_position_embeddings": 131072
}
vllm serve Qwen/Qwen3-8B \
--max-model-len 131072 \
--rope-scaling '{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}'
Does static YaRN hurt short prompts too?
Yes, and this is the part people miss. Static YaRN applies the same interpolation to every request. A 400-token prompt gets its slow dims compressed by 4× for no reason — you paid the long-context tax on traffic that never needed it. Qwen's own docs say as much: turn YaRN on only when long inputs are actually expected.
Dynamic scaling computes s = max(1, L_current / L_orig) per sequence, so short prompts run the identity transform. The cost is that the frequency table changes as a sequence grows, meaning keys already in the cache were rotated under a different s than new ones. Implementations paper over this in various ways, but it complicates clean prefix-cache reuse, which is exactly the optimization you want on long prompts. If you serve mixed traffic, the cheap answer is two deployments: an unscaled one for the short high-QPS path, a YaRN one for the long path, routed on token count.
How do I test whether my RoPE change hurt short context?
Needle tests are not enough — they measure retrieval, not positional resolution, and they only probe the long regime. Bucket your eval by prompt length and report deltas per bucket, with 0-2k as its own line. Then add at least one task that requires ordering rather than presence: reproduce the k-th item from a list, identify which of two near-identical spans came first, or apply numbered rules from a long system prompt in order. Those are the tasks that read the mid-frequency dims you just stretched.
For hosted models — Claude Opus 4.x, Sonnet 4.x, GPT-5.x — none of this is a knob you own. The provider ships one trained configuration. The corollary still applies though: keep prompts as short as the task allows, because you are always somewhere on this trade-off curve, whoever set the dial.
So why does raising rope_theta break short-context accuracy?
Because rope_theta sets a bank of positional wavelengths, λ_i = 2π · base^(2i/d), and raising it stretches the mid and high dims far more than the fast ones. Dimensions that used to resolve position within a few hundred tokens now need thousands, so inside an ordinary prompt they emit a near-constant signal instead of a positional one — at d=128, base 1e4 → 1e6 cuts the dims with sub-2048-token wavelength from 40 to 27. The attention heads were trained against the old allocation and were never retrained for the new one. Fine-tuning at the new base fixes it; an inference-time config edit does not. If you need the length without retraining, use YaRN, which interpolates only the dims that never completed a full rotation during training and leaves the adjacency clocks alone — and turn it on per request rather than globally.
Top comments (0)