A certified classifier comes with a proof for each input: inside a radius r around the image, no perturbation can change the predicted class. In medical imaging that radius has to cover what happens in a clinic every day, such as a second photo of the same patch of skin taken at a slightly different angle.
The proof rests on the Lipschitz constant L of the network. The certificate divides the logit margin by L, so when fine-tuning doubles L, the radius at the same margin halves. Certified networks are trained on ImageNet while a clinic has a few thousand labelled images, so in practice you fine-tune, and in our experiments LoRA raised L by 1.9 to 2.5 times and full fine-tuning by 5 to 8.5 times.
A rotation keeps every singular value of the weights
The butterfly adapter multiplies each frozen weight matrix by an orthogonal matrix, W′ = R(θ)·W. For orthogonal R, (RW)ᵀ(RW) = WᵀW, so W′ has the singular values of W and its spectral norm stays where it was, whatever training does to θ.
R(θ) is a product of Givens rotations laid out as a butterfly: log₂ n levels, each of which pairs up the channels and rotates every pair by its own angle. A Givens rotation is orthogonal for any angle, so Adam trains the angles directly with no projection step or penalty term. Acting on unary states, the same rotation is a reconfigurable beam splitter (RBS) gate, which makes each adapter a quantum circuit of depth log₂ n. We simulate it on a GPU as an ordinary matrix.
A 2048 wide layer holds about 4.2 million weights, and its butterfly has 11 264 angles.
Check the bound in 20 lines
The layer ships in eigenkit, our open-source library (intro post). The script runs on a CPU in a few seconds.
pip install eigenkit==1.2.2 torch
import torch
from eigenkit.orthogonal.torch import Orthogonal, rotate_weight
torch.manual_seed(0)
n = 2048
W = torch.randn(n, n) / n**0.5 # a frozen pretrained weight
L0 = torch.linalg.matrix_norm(W, ord=2) # its spectral norm (Lipschitz bound)
# butterfly adapter: R(theta) @ W, R orthogonal for every theta
R = Orthogonal(n) # butterfly layout of RBS / Givens gates
with torch.no_grad():
for p in R.parameters():
p.normal_(0, 1.0) # any angles at all
W_bfly = rotate_weight(R.matrix(), W)
# LoRA, rank 4: W + B @ A, random factors standing in for trained ones
A = torch.randn(4, n) * 0.05
B = torch.randn(n, 4) * 0.05
W_lora = W + B @ A
print("butterfly angles:", sum(p.numel() for p in R.parameters()))
print("LoRA params: ", A.numel() + B.numel())
print("L/L0 butterfly: ", f"{(torch.linalg.matrix_norm(W_bfly, ord=2) / L0).item():.3f}")
print("L/L0 LoRA: ", f"{(torch.linalg.matrix_norm(W_lora, ord=2) / L0).item():.3f}")
butterfly angles: 11264
LoRA params: 16384
L/L0 butterfly: 1.000
L/L0 LoRA: 2.764
Any angles you draw give a ratio of 1 up to float32 rounding. The LoRA ratio depends on its factors: the random ones here only show the mechanism, and the trained LoRA adapters in the paper ended at 1.9 to 2.5.
Results on trichoscopy images from a clinic
With dermatologists at the University of Trieste we adapted BRONet, a certified ImageNet network, to trichoscopic scalp images for two tasks: diagnosis of hair loss across 5 classes (3 581 images from 143 patients) and activity of alopecia areata (1 196 images from 57 patients). The adapter acts on the 8 dense layers of BRONet's MLP, and every method trains a new classification head.
Diagnosis task, 717 test images:
| Butterfly | LoRA | Full fine-tuning | |
|---|---|---|---|
| Certified accuracy, ε = 36/255 | 63.8% | 31.6% | 35.6% |
| Certified accuracy, ε = 108/255 | 20.4% | 0% | 0% |
| Clean accuracy | 80.4% | 71.5% | 82.5% |
| L / L₀ after adaptation | 1.00 | 1.9 to 2.5 | 5 to 8.5 |
| Trainable parameters | about 100 000 | 141 000 | 33.6 million |
Training only the new head also keeps L unchanged and reached 55.2% certified accuracy, so the butterfly adds 8.6 points by adapting the features as well. On the alopecia areata activity task the butterfly certified 40.6% of images, against 21.3% for LoRA and 17.9% for full fine-tuning.
We also attacked the adapted models with PGD and APGD-T. No attack found an adversarial example inside a certified radius, and the certified radius tracked the smallest successful attack on each image with a Spearman correlation of 0.97 to 0.99.
Paper and code
- Paper: Quantum-inspired orthogonal adapters for certifiably robust medical image classification, arXiv:2611.18775
- Code that reproduces every table: github.com/eigenorbit/quantum-inspired-fine-tuning
- The orthogonal layers, the certificate and the losses: eigenkit on GitHub, module
eigenkit.orthogonal.torch
If you fine-tune a Lipschitz-constrained model on your own data, swapping LoRA for the butterfly takes two calls in the training loop: Orthogonal(n) for each frozen layer and rotate_weight(R.matrix(), W) in its forward pass.
Top comments (0)