A trained checkpoint is nothing but a very long list of numbers. Two models fine-tuned from the same base therefore live in the same weight space, at the same coordinates, and you can do arithmetic on them. That single observation is the whole of model merging — and it is how a large share of the open-weight models on Hugging Face leaderboards are actually made.
The alternative is retraining on the union of both datasets: GPUs, data access, hours to days. Merging needs no training data, runs on CPU, and takes minutes.
Task vectors: a skill is a direction
Subtract the base from a fine-tune and everything the base already knew cancels out. What's left is only what fine-tuning changed:
tau_a = {k: ft_a[k] - base[k] for k in base} # the "sentiment" direction
tau_b = {k: ft_b[k] - base[k] for k in base} # the "spam" direction
# reconstruction is exact by construction
assert torch.allclose(base[k] + tau_a[k], ft_a[k])
That difference is the task vector, and the striking empirical fact is that it behaves like an object you can manipulate. It points in a consistent direction, its length controls how strongly the skill is expressed, and different tasks give roughly orthogonal vectors. Most of its entries are near zero — fine-tuning moves very little. That near-orthogonality is exactly why one model can carry two skills at once.
One hard requirement follows from the geometry: the models must share a pre-trained ancestor, plus the same architecture and tokenizer. Two models trained from scratch with different seeds can compute the identical function while having wildly different weights — neurons permuted, signs flipped — so averaging them produces noise. Fine-tunes of one base start at the same point and stay in the same low-loss basin, which is the property (linear mode connectivity) that makes any of this work.
The one-line merge, then the dial
The crudest merge is a plain parameter-wise average, the model soup:
def soup(models, weights=None):
w = weights or [1 / len(models)] * len(models)
return {k: sum(wi * m[k] for wi, m in zip(w, models)) for k in models[0]}
Written in task-vector form that is base + 0.5*tau_a + 0.5*tau_b — task arithmetic with the coefficients forced to sum to one. Surprisingly often it is already most of the win. In the demo below, a sentiment classifier scores 100% on its own task and 32% on spam; a spam filter scores 100%/42% the other way. Their soup scores 97% and 98% — one set of weights, both jobs.
Give each vector its own coefficient and you get a mixing desk:
def task_arithmetic(base, taus, lambdas):
return {k: base[k] + sum(l * t[k] for t, l in zip(taus, lambdas))
for k in base}
Lambda is the single most important knob, and it is not free. Push both to 1.0 in the demo and accuracy falls to 89% — below the 0.5 soup — because the two vectors' off-task noise now stacks instead of averaging. The sweep peaks near lambda = 0.4 at 99.2%. Every real merge recipe ends with a hyper-parameter search against a held-out eval.
Negation: subtract to unlearn
Because a skill is a direction, reversing the sign walks the model away from it:
forget = {k: base[k] - 1.0 * tau_toxic[k] for k in base}
In the demo this pushes sentiment accuracy from 50% down to 20% — below a coin flip, because the model hasn't merely forgotten the task, it has been handed the anti-skill and now predicts the opposite label. The other task doesn't move at all. The same trick is used seriously for detoxification, stripping a memorised style, or removing a capability from a released checkpoint. It is by far the cheapest form of unlearning, though it's a blunt instrument: large negative lambda damages general ability too.
Interference is the real problem
Merging fails in two specific ways.
Redundancy — the vast majority of a task vector's entries are noise-sized. They contribute nothing but dilute the entries that matter once you average.
Sign conflict — on parameters both skills care about, they often want opposite directions. In the demo the word "service" gets +0.42 from the sentiment vector (praise in a review) and -0.73 from the spam vector (routine in an inbox). Average those and you get -0.15: a value neither model wanted, weak enough that both skills lose that parameter.
That is why merging more and more models keeps getting worse, and it's what the next three methods each attack from a different angle.
TIES: Trim, Elect, Disjoint Merge
TIES fixes both problems in three passes.
stack = torch.stack([t[k].flatten() for t in taus]) # [n_models, P]
# 1. TRIM - keep top-k% by magnitude, per model
keep = int(k * stack.shape[1])
thresh = stack.abs().kthvalue(stack.shape[1] - keep + 1, dim=1, keepdim=True).values
stack = stack * (stack.abs() >= thresh)
# 2. ELECT - one sign per parameter, by total magnitude
elected = stack.sum(0).sign()
# 3. DISJOINT MERGE - mean over the entries that agree
agree = (stack.sign() == elected) & (stack != 0)
merged = (stack * agree).sum(0) / agree.sum(0).clamp(min=1)
Trim deletes the redundancy. Elect makes one direction win outright instead of being averaged away. Disjoint merge averages only the models agreeing with the elected sign, so the winner keeps its full magnitude — that contested service parameter comes out at -0.73, not -0.15. In the demo TIES is the best merge on the board at 99.2%, beating the plain soup, and unlike averaging it keeps scaling as you add models. Trim too hard, though, and you start deleting real signal: at k = 20% the demo drops to 83%.
DARE: drop 90%, rescale, lose nothing
DARE takes the redundancy observation to its extreme — delete a random fraction of each task vector, then rescale the survivors so the expected vector is unchanged:
def dare(tau, p=0.9):
mask = torch.rand_like(tau) >= p # keep each entry w.p. 1-p
return tau * mask / (1 - p) # rescale so E[.] is unchanged
You can drop 90–99% of a fine-tune's delta and the model still performs, which is a remarkable statement about how little fine-tuning actually changes. For merging the benefit is geometric: two vectors that are each 10% dense collide on roughly 1% of parameters instead of 100%, so interference nearly vanishes. Note the drop is random, not magnitude-based — that's the difference from TIES trimming — and the two compose, which mergekit exposes as dare_ties.
SLERP: walk the arc, not the chord
A straight line between two weight vectors pointing in different directions passes through the sphere they sit on, so the linear average is shorter than either parent and every activation shrinks. SLERP walks the arc at constant angular speed and interpolates the magnitude separately:
def slerp(a, b, t=0.5):
na, nb = a.norm(), b.norm()
ua, ub = a / na, b / nb
omega = torch.acos((ua * ub).sum().clamp(-1, 1))
s = torch.sin(omega)
direction = (torch.sin((1 - t) * omega) * ua + torch.sin(t * omega) * ub) / s
return direction * ((1 - t) * na + t * nb)
Norm preserved, and at t = 0 or 1 you recover a parent exactly. It handles exactly two models; for three or more, use TIES or DARE.
In practice
Nobody hand-rolls this at 7B scale. mergekit takes a YAML recipe, streams tensors so it runs on CPU with modest RAM, and writes a normal checkpoint you can serve anywhere:
base_model: mistralai/Mistral-7B-v0.1
merge_method: ties # linear | task_arithmetic | ties | dare_ties | slerp
dtype: bfloat16
models:
- model: org/mistral-sentiment
parameters: {weight: 0.5, density: 0.5} # weight = lambda, density = top-k
- model: org/mistral-spam
parameters: {weight: 0.5, density: 0.5}
The merge itself takes minutes. The real work is the sweep — lambda, density, t are all hyper-parameters, and merging cannot create a skill neither parent had. Producing candidates is cheap, which makes evaluating them the actual job. A merge is a hypothesis; the eval set is the answer.
The demo really does train a base and two fine-tunes in your browser by gradient descent, then merges them live — drag the sliders and watch held-out accuracy on both tasks move: https://dev48v.infy.uk/ai/days/day59-model-merging.html
Top comments (0)