SFT vs. RL: What Changes Inside the Model?

In modern generative AI post-training, two fundamental paradigms dominate: Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR / GRPO).

While practitioners often treat SFT and RL as interchangeable steps on an incremental tuning ladder, they perform mathematically and mechanistically distinct operations on a model’s internal parameters:

• SFT is Knowledge Injection: It uploads new factual domain knowledge, API syntax, vocabulary associations, and conversational formatting into the model’s representation space.
• RL is Reasoning, Search & Routing: It does not inject new facts. Instead, it teaches the model how to systematically explore, evaluate, backtrack, and route its existing pre-trained capabilities to solve verifiable multi-step problems.

Recent breakthrough work by Zhu et al. (July 2026, arXiv:2607.19331, “ISO: An RLVR-Native Optimization Stack”) provides rigorous mathematical proof of this dichotomy through the lens of Singular Value Decomposition (SVD): during RLVR post-training, the singular value spectrum (Σ) of transformer weight matrices remains virtually unchanged from the base model. RL adaptation occurs almost entirely through rotations of the singular coordinate frames (U and V).

Start here

Explore how a weight matrix is decomposed into three matrices (U, Σ, Vᵀ), why SFT rewrites the diagonal spectrum to upload new facts, and why RLVR freezes the spectrum while rotating coordinate frames to master multi-step reasoning.

• SVD Decomposition: Factoring a weight matrix W into U · Σ · Vᵀ (output frame, singular values spectrum, input frame).
• Spectrum (Σ): A diagonal matrix of singular values representing the energy and capacity allocated to concepts in the model.
• Singular Frames (U, V): Orthonormal coordinate systems that detect incoming features (Vᵀ) and route outputs to the next layer (U).
• Spectral Inheritance: The empirical discovery that reinforcement learning preserves the base model's singular spectrum (Σ ≈ Σ₀).

SFT grows the singular values Σ to store new facts. RLVR keeps Σ ≈ Σ₀ and rotates the frames U and Vᵀ instead.

The original post has an interactive 3D version of this diagram.

1. The Geometry of a Weight Matrix: SVD Decomposition

Any weight projection tensor in a transformer layer—whether in self-attention (W_q, W_k, W_v, W_o) or feed-forward MLP projections (W_gate, W_up, W_down)—can be factored into three fundamental linear algebraic components via Singular Value Decomposition:

$$W = U \cdot \Sigma \cdot V^{\mathsf T}$$

What W = U·Σ·Vᵀ does: rotate the input (Vᵀ), stretch each direction by its singular value (Σ), rotate again (U).

Where each of the three matrices governs a distinct physical role in representation learning:

1. Right Singular Vectors (Vᵀ, the Input Frame): Functions as the model’s input feature detector. It defines an orthonormal coordinate system that maps incoming token representations from context space into the model’s internal semantic subspaces.
2. Singular Values (Σ, the Diagonal Spectrum): A non-negative diagonal matrix containing singular values σ₁ ≥ σ₂ ≥ … ≥ σᵣ ≥ 0. The spectrum dictates the energy, scaling power, and intrinsic capacity allocated to each concept. This represents the model’s fundamental knowledge storage.
3. Left Singular Vectors (U, the Output Frame): The output semantic projection. It maps features out of the latent space and routes them to the next transformer layer or the final vocabulary unembedding projection.

2. What SFT Actually Does: Rewriting the Spectrum (ΔΣ >> 0)

When you train a model with Supervised Fine-Tuning on next-token cross-entropy loss:

$$\mathcal{L}{\mathrm{SFT}}(\theta) = -\sum{t=1}^T \log P_\theta(y_t \mid y_{<t}, x)$$

The objective forces the model to memorize target token distributions. If an enterprise wants a model to learn proprietary Verilog hardware libraries, internal legal compliance rules, or specialized medical terminology:

• New Concepts Require New Energy: The model must amplify representations along directions that were previously suppressed or dormant during pre-training.
• Spectrum Rescaling: SFT actively alters the diagonal matrix Σ. New singular values expand, existing ones shift, and the model’s spectral energy distribution is rewritten to store the injected domain knowledge.
• Frame Adaptation: Concurrently, U and Vᵀ rotate to recognize new prompt formats and emit specific stylistic tokens.

Because SFT rewrites Σ, it is prone to catastrophic forgetting: over-amplifying new singular values can collapse or distort pre-trained reasoning circuits.

3. What RL Does: Spectral Inheritance (arXiv:2607.19331)

In reinforcement learning with verifiable rewards (such as GRPO applied to competitive math, coding, or formal verification), the loss is driven by outcome verification:

$$\mathcal{L}{\mathrm{RLVR}}(\theta) = \mathbb{E}\left[ \frac{\pi\theta(a \mid s)}{\pi_{\mathrm{old}}(a \mid s)} \cdot A(s, a) \right]$$

In their landmark paper “ISO: An RLVR-Native Optimization Stack”, Zhu et al. (UT Austin, Together AI) discovered a surprising empirical property: Spectral Inheritance.

$$\Sigma_{\mathrm{RLVR}} \approx \Sigma_0$$

Why Does RL Preserve the Spectrum?

Reinforcement learning with outcome rewards cannot invent facts out of thin air. When an agent is rewarded for solving a multi-step geometry proof or finding a SQL join error, the reward signal does not provide external factual knowledge.

Instead, RL solves a routing and verification problem:

• The underlying representation capacity (Σ₀) was already acquired during pre-training on trillions of tokens.
• RL optimizes the input singular frame V to better isolate critical problem constraints.
• RL rotates the output singular frame U to connect intermediate thinking steps to high-probability verification circuits (backtracking, error detection, self-correction).

In linear algebraic terms, reasoning is a rotation of coordinate frames, not an expansion of spectral energy.

4. Isospectral Optimization (ISO): Training on Frames Alone

Zhu et al. turned this insight into a principled post-training paradigm called Isospectral Optimization (ISO):

1. ISO-Optimizer (Online RL): During training, freeze the base model’s singular values Σ₀ permanently. Run optimizers (like AdamW or Muon) exclusively over the frame variables (U, V):

$$W(t) = U(t) \cdot \Sigma_0 \cdot V(t)^{\mathsf T}$$

On Qwen3-8B, standard AdamW required 270 steps to reach 0.495 aggregate accuracy. ISO-AdamW reached that same accuracy in only 100 steps (2.7x fewer steps) and continued improving to 0.509 at 210 steps, demonstrating that constraining optimization to the true manifold of reward-driven adaptation speeds up training.

2. ISO-Merger (Offline Specialist Composition): If two specialist models (e.g., a math agent and a coding agent) share the same base spectrum Σ₀, their learned capabilities live exclusively in their frame changes ΔU and ΔV. ISO-Merger composes these specialists directly in frame space without requiring rollouts, gradients, or distillation data.

Empirical Update (September 2026): Does ISO beat AdamW in practice? Following the ISO paper’s equations, we implemented ISO-AdamW and benchmarked it head-to-head against standard AdamW on a dedicated NVIDIA H200 GPU using GRPO on 1,000 held-out GSM8K math problems (Qwen3-1.7B-Base). While ISO-AdamW reached 75.8% accuracy versus AdamW’s 75.4% (+4 answers out of 1,000), it required +47.5% more peak VRAM and intensive polar projection iterations. Read our full end-to-end empirical report and systems post-mortem: We Tried ISO-AdamW. AdamW Kept Its Job →

5. Comparing SFT vs. RL Post-Training

Dimension	Supervised Fine-Tuning (SFT)	Reinforcement Learning (RLVR / GRPO)
Primary Function	Knowledge injection & style imitation	Reasoning, search exploration & verification
What the Model Learns	“What to say” (facts, syntax, vocabulary)	“How to think” (search trees, backtracking, error recovery)
Weight Spectrum (Σ)	Rewritten (ΔΣ >> 0) — allocates new singular values	Inherited (Σ ≈ Σ₀) — remains near-constant
Singular Frames (U, V)	Adapted to target surface style	Heavily rotated to route logic into verifiers
Risk Mode	Catastrophic forgetting of pre-trained reasoning	Reward hacking or policy collapse if unconstrained
Data Requirement	High-quality input-output demonstration pairs	Stateful environments with deterministic verifiers

6. Practical Implications for Enterprise AI Systems

Understanding this mathematical boundary changes how enterprise post-training pipelines should be built:

• Never Expect RL to Teach Unknown Facts: If your base model has never seen your proprietary CAD kernel API, Modelica equations, or private database schemas, running GRPO will fail. Because RL cannot expand spectral capacity (Σ), it cannot discover facts out of vacuum.

The Governed Two-Stage Pipeline:

1. Stage 1 (SFT): Upload domain knowledge, grammar, and demonstration traces. This expands the spectrum Σ to encompass the domain vocabulary.
2. Stage 2 (RLVR in OpenEnv): Connect the adapted model to executable gyms with deterministic verifiers (lint, compiler, simulation). RL realigns the singular frames (U, V) to master reasoning, trade-off evaluation, and self-correction.

---

Originally published at g-ftech.com.