A block-pruning method published today reframes which transformer layers to remove as a constrained binary optimization problem mapped onto an Ising spin glass, rather than scoring each block independently. Tested on Llama-3.3-70B-Instruct at 50% depth compression, it holds MMLU at 76.9 without any retraining, against 54.0 for a standard block-importance baseline. The code is open source, which means teams already doing depth pruning for inference cost savings can test the claim directly rather than take it on faith.
What changed
Researchers at Multiverse Computing released a new approach to transformer block pruning that treats the choice of which layers to keep as a constrained binary optimization (CBO) problem rather than a per-block scoring exercise. The method performs a second-order Taylor expansion of the model's loss around its current weights, producing a Hessian matrix over the set of blocks. Diagonal entries of that Hessian capture how much removing a single block alone would hurt performance -- the same information most existing pruning methods rely on -- but the off-diagonal entries capture how pairs of blocks interact when removed together, which single-block scoring throws away entirely.
That Hessian maps directly onto the energy function of an Ising spin glass, where each block is a spin (kept or removed) and the off-diagonal terms become spin-spin couplings. Finding a low-energy spin configuration then becomes a proxy for finding a pruned model that will score well, and the team solves this with standard combinatorial optimization techniques rather than any physics-specific hardware.
Results
The team tested four models -- Llama-3.1-8B-Instruct, Qwen3-14B, Llama-3.3-70B-Instruct, and the hybrid Mamba2/attention/MoE NVIDIA-Nemotron-3-Nano-30B -- across MMLU, AIME25, and GPQA. The headline result removes 40 of Llama-3.3-70B-Instruct's 80 blocks (50% depth compression) and reaches 76.9 on MMLU with zero retraining, against 54.0 for a standard block-influence baseline -- a 23-point gap at the same compression ratio.
One counterintuitive finding: after retraining, some higher-energy ('excited state') spin configurations outperformed the lowest-energy solution the optimizer found, suggesting the energy landscape correlates with post-pruning quality but doesn't fully determine it. The implementation is open-sourced at CompactifAI/Block_removal_through_constrained_binary_optimization on GitHub.
Who this affects
Teams running large open-weight models (70B and up) who need to cut inference cost through depth pruning are the direct audience -- this method's main claim is retaining far more accuracy at the same compression ratio than the block-importance scoring most existing pruning tools use, with no retraining required to see the initial gain. It matters most at aggressive compression ratios (50% and beyond), where single-block scoring methods degrade sharply because they ignore how removed blocks interact with each other.
It's less relevant for teams doing light pruning (10-20%) where baseline methods already perform adequately, or for teams working with models under roughly 8B parameters where the four tested model sizes don't provide direct evidence. Teams using architectures outside the tested set -- pure Mamba or pure MoE without the hybrid pattern tested here -- should treat the results as suggestive rather than confirmed for their specific architecture.
Verdict
Adopt this for evaluation, not production, today. The reported numbers come from the authors' own benchmarks without independent replication, and the method's core claim -- that pairwise block interactions matter more than single-block importance -- is exactly the kind of result that benefits from a second team reproducing it on a different model family before teams bet inference infrastructure on it.
That said, the bar to test it yourself is low: the code is open source, the method requires no retraining to get an initial pruned model, and it slots in as a drop-in alternative to whatever block-importance scoring a team's existing pruning pipeline already uses. Teams currently accepting the accuracy hit from standard block-importance pruning at 50%+ compression have little to lose by running this against their own model and benchmark suite before deciding whether to switch.
Tracked daily from official release feeds and vendor changelogs. Full archive: https://media.patentllm.org
Top comments (0)