rsLoRA: Why LoRA alpha/r Scaling Kills High-Rank Fine-Tunes

You bumped r from 16 to 256, expecting the adapter to soak up more of the task. Training took 4x longer, VRAM went up, and the eval loss landed within noise of the rank-16 run. The usual explanation — "your task is low-rank, 16 was already enough" — is often wrong. The real culprit is LoRA alpha/r scaling: the default gamma = alpha / r shrinks the adapter's effective update as rank grows, so a rank-256 adapter takes systematically smaller steps than a rank-8 one at the same learning rate.

rsLoRA (rank-stabilized LoRA) replaces that divisor with sqrt(r). It's a one-line config flag, and it is one of the few one-line flags in fine-tuning that changes the shape of your rank sweep instead of nudging a number.

TL;DR

• LoRA computes W' = W + (alpha/r) · B·A. The 1/r divisor is not a normalizer — it actively decays the magnitude of the learned update as rank increases.
• Under AdamW, the Frobenius norm of the effective ΔW after a fixed number of steps falls roughly as 1/sqrt(r) with classic scaling, and stays flat with alpha/sqrt(r).
• Consequence: rank sweeps at fixed alpha are learning-rate sweeps in disguise. High ranks look useless because they are undertrained, not because they are overparameterized.
• Fix: use_rslora=True in PEFT. To keep your current effective scale when switching at rank r, set alpha_rs = alpha_old / sqrt(r) — otherwise you silently multiply your step size by sqrt(r).
• use_rslora lives in adapter_config.json. Rebuild the config by hand at load or serve time and you load the same tensors at the wrong scale — a quiet, non-crashing accuracy regression.

What does the alpha/r scaling factor actually do?

LoRA freezes W ∈ R^(d_out × d_in) and learns a low-rank correction:

y = W·x + gamma · B·(A·x)
A ∈ R^(r × d_in),  B ∈ R^(d_out × r),  gamma = alpha / r

A gets Kaiming init, B is zeroed, so the adapter contributes nothing at step 0 and the merged model equals the base model. That part is fine.

The folklore is that alpha/r "keeps the scale constant when you change rank, so you don't have to retune the learning rate." That claim is doing a lot of unearned work. It would hold if B·A grew linearly in r. It doesn't.

Why does LoRA alpha/r scaling break high-rank adapters?

Because the product B·A grows sublinearly in r, so dividing by r overcorrects. Work through the first optimizer step.

With B = 0, the gradient to A is exactly zero — A cannot move until B does. So the first thing that happens is:

dL/dB = gamma · g · (A·x)ᵀ        where g = dL/dy

Under plain SGD with rate eta, B₁ = -eta·gamma·g·(A·x)ᵀ, and the resulting change in the effective weight is

ΔW = gamma · B₁ · A = -eta · gamma² · g · (A·x)ᵀA

The term (A·x)ᵀA = Σᵢ (A·x)ᵢ · A[i,:] is a sum of r terms. Its component along x is Σᵢ (A·x)ᵢ² / ‖x‖ — every summand is positive, so that part adds coherently and grows like r. Total: ΔW ~ eta · gamma² · r.

Plug in gamma = alpha/r and you get ΔW ~ eta · alpha² / r. The update vanishes as 1/r. Plug in gamma = alpha/sqrt(r) and you get ΔW ~ eta · alpha² — rank-independent. That is the entire rsLoRA argument.

Does AdamW cancel the problem?

No, but it changes the exponent. AdamW normalizes per-parameter, so the size of ΔB is set by the learning rate rather than by the gradient magnitude, and the gamma² term collapses to gamma¹:

ΔB ≈ eta (element-wise, gamma-independent)
ΔW = gamma · ΔB · A

Now ΔB·A sums r largely incoherent contributions (Adam's sign-like updates don't align with A's rows the way the SGD case does), so it grows like sqrt(r):

classic:  ΔW ~ (alpha/r)·eta·sqrt(r)      = alpha·eta / sqrt(r)   → decays
rsLoRA:   ΔW ~ (alpha/sqrt(r))·eta·sqrt(r) = alpha·eta            → flat

Same conclusion, milder slope. Going from r=8 to r=256 is a 32x rank increase, so classic scaling costs you about sqrt(32) ≈ 5.7x in effective update magnitude. That is not a rounding error; it's the difference between converging and crawling.

You can measure this in twenty lines. No model needed — the effect is a property of the parameterization:

import math, torch
from torch import nn

D_IN, D_OUT, LR, ALPHA, STEPS = 2048, 2048, 1e-4, 16.0, 20

def delta_w_norm(rank: int, mode: str) -> float:
    torch.manual_seed(0)
    gamma = ALPHA / rank if mode == "lora" else ALPHA / math.sqrt(rank)
    A = nn.Parameter(torch.empty(rank, D_IN))
    nn.init.kaiming_uniform_(A, a=math.sqrt(5))
    B = nn.Parameter(torch.zeros(D_OUT, rank))
    opt = torch.optim.AdamW([A, B], lr=LR, weight_decay=0.0)
    for _ in range(STEPS):
        x = torch.randn(32, D_IN)
        target = torch.randn(32, D_OUT)
        loss = ((gamma * (x @ A.T) @ B.T - target) ** 2).mean()
        opt.zero_grad(); loss.backward(); opt.step()
    return (gamma * B @ A).norm().item()      # ‖ΔW‖_F actually applied to the base weight

for r in (8, 16, 64, 256):
    print(f"r={r:4d}  lora={delta_w_norm(r,'lora'):.4f}  rslora={delta_w_norm(r,'rslora'):.4f}")

The lora column falls off roughly as 1/sqrt(r); the rslora column is close to flat. Identical learning rate, identical steps, identical alpha. The only thing that changed is how much of the learned update actually reaches W.

How do I turn on rsLoRA in PEFT?

One flag. It changes the scaling factor only — the tensors, shapes, and merge math are untouched:

from peft import LoraConfig, get_peft_model

cfg = LoraConfig(
    r=128,
    lora_alpha=16,          # NOT 2*r — see the migration note below
    use_rslora=True,        # gamma = lora_alpha / sqrt(r)
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(base, cfg)

The flag is serialized:

{
  "r": 128,
  "lora_alpha": 16,
  "use_rslora": true,
  "target_modules": ["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"]
}

How do I migrate alpha without blowing up the run?

Match the effective scale first, then hold alpha fixed while you sweep rank. Setting gamma_rs = gamma_old:

alpha_rs / sqrt(r) = alpha_old / r   →   alpha_rs = alpha_old / sqrt(r)

current r	current alpha	gamma	equivalent rsLoRA alpha
8	16	2.0	5.66
16	32	2.0	8
64	128	2.0	16
256	512	2.0	32

Flip use_rslora=True and leave alpha alone at r=256 and your effective scale jumps 16x. Since B starts at zero, you won't see a spike at step 0 — you'll see a loss curve that looks fine for a hundred steps and then diverges or flatlines into a degenerate output distribution. People misread this as "rsLoRA is unstable." It isn't; you changed your step size by an order of magnitude.

alpha is not a learning rate, but at fixed rank it multiplies the adapter's contribution linearly, so it behaves like one for the adapter branch. Retune lr and alpha together, not independently.

Why do rank sweeps at fixed alpha lie?

Because with classic scaling, changing r at fixed alpha changes gamma by 1/r, so you're varying two things at once. The two common sweep protocols both fail differently:

• Fixed alpha (e.g. alpha=16 across r ∈ {8…256}): gamma drops 32x across the sweep. High ranks are catastrophically undertrained. You conclude "rank doesn't matter."
• alpha = 2r: gamma is pinned at 2.0 for every rank, which looks controlled. But the effective update still decays like 1/sqrt(r) under Adam because B·A itself grows. You conclude "rank saturates around 16-32."

Under rsLoRA with fixed alpha, the effective update magnitude is roughly rank-independent, so a rank sweep measures capacity instead of step size. This is the practical payoff: you finally get a monotone-ish curve where higher rank helps until the task genuinely runs out of signal, and you can see where that point actually is.

A complementary fix worth knowing: LoRA+ assigns B a higher learning rate than A (PEFT exposes this through create_loraplus_optimizer, typically with a ratio in the 8-16 range). It targets the same asymmetry — B starts at zero and gates everything — from the optimizer side rather than the scaling side. They compose, but change one at a time.

What breaks at merge and serve time?

Scale mismatch, silently. use_rslora is a property of the config, not the weights. The safetensors file holds A and B; nothing in it records which divisor produced them. So:

• Reconstructing a LoraConfig by hand in an eval script instead of loading adapter_config.json drops the flag. You then apply alpha/r to weights trained under alpha/sqrt(r) — at r=128 that's an 11x under-scaled adapter. The model loads, generates fluent text, and quietly behaves closer to the base model.
• Any serving stack that parses adapter_config.json itself for multi-LoRA hot-swapping needs to honor use_rslora. Verify it rather than assuming: load the adapter, run one prompt through the served path and the same adapter merged locally with merge_and_unload(), and compare logits. If the scale is wrong, the divergence is immediate and obvious.
• merge_and_unload() bakes gamma into W, so merged checkpoints are safe to ship anywhere. If you have mixed tooling, merging is the robust move.

Add a one-line assert in your eval harness: load the adapter and check peft_model.peft_config["default"].use_rslora matches what training wrote. It costs nothing and catches the whole class of failure.

When is classic alpha/r still fine?

At r ≤ 16 with a tuned learning rate, the difference is small and you've already absorbed it into lr. If you have a working rank-8 recipe, leave it. rsLoRA matters when you want rank to be a real, tunable capacity knob — domain adaptation with substantial new vocabulary or style, multi-task adapters, or any setting where you suspect the task isn't low-rank and want to test that honestly.

The short answer

LoRA alpha/r scaling divides the adapter output by r, but the product B·A only grows like sqrt(r) under AdamW, so the update that actually reaches your frozen weights decays as 1/sqrt(r) — a rank-256 adapter trains at roughly a sixth the effective step size of a rank-8 one at identical hyperparameters. High rank appears not to help because it never got trained, not because the task is low-rank. rsLoRA (use_rslora=True) swaps the divisor for sqrt(r), making the effective update magnitude rank-independent. When you switch, rescale alpha by 1/sqrt(r) to preserve your current step size, then hold alpha constant across the sweep — and make sure every load and serve path reads use_rslora from adapter_config.json, because getting it wrong costs accuracy without throwing an error.