The Lottery Ticket Hypothesis: Why Sparse Subnetworks Train as Well as Dense Ones

Large neural networks are usually trained dense and pruned afterwards. The lottery ticket hypothesis turns that order on its head. It claims that inside a large, randomly initialised network there already exists a small subnetwork - a winning ticket - which, when trained on its own from its original initialisation, can reach accuracy comparable to the full network in a similar number of training steps.

That is a strong claim. It suggests that much of training is not about finding good weights from scratch, but about finding, and then training, a lucky subnetwork that was present from the start. It also suggests that overparameterisation helps partly because it buys more tickets in the lottery.

The intuition: why would a subnetwork be special?

Consider training a dense network. Most weights change, many become small, and pruning the smallest weights after training often barely hurts accuracy. A natural reading is that the pruned weights were never needed. A sharper reading is that the remaining weights were already well placed at initialisation, and training mostly refined them.

The lottery ticket hypothesis makes the sharper reading testable. Take the trained network, keep only a subset of its weights, reset those weights to the values they had at initialisation, and retrain only that subset. If the retrained sparse network matches the dense network, then the structure of the subnetwork, together with its initial weights, carried most of the information. If it does not, then the dense training trajectory itself mattered in a way that cannot be recovered from the initial weights alone.

This distinction matters for practice. Pruning after training gives you a smaller model for inference, but you still paid the full cost of dense training. A true winning ticket would let you train sparse from the start, saving compute throughout training, not just at deployment.

Formalising a ticket

Let a dense network be written as

$$f(x; \theta)$$

where

$\theta \in \mathbb{R}^{d}$

is the vector of parameters and

$d$

is large. A subnetwork is defined by a binary mask

$m \in {0, 1}^{d}$

applied elementwise:

$$f(x; m \odot \theta)$$

Here

$\odot$

denotes elementwise multiplication, so a mask entry of zero removes the corresponding connection entirely.

Let

$\theta_{0}$

be the initialisation and

$\theta_{T}$

be the parameters after training the dense network. A winning ticket is a mask

$m$

such that training

$f(x; m \odot \theta_{0})$

from scratch reaches test accuracy comparable to

$f(x; \theta_{T})$

in a comparable number of iterations. The sparsity of the ticket is the fraction of mask entries equal to zero. A ticket that keeps only a small fraction of the weights, yet trains to full accuracy, is a strong result. A ticket that keeps almost everything is not interesting, because it is barely different from the dense network.

Two details in this definition do a lot of work. First, the ticket is trained from

$\theta_{0}$ ,

not from random new weights and not from

$\theta_{T}$ .

Reinitialising the subnetwork destroys the effect in the original experiments, which tells us that the specific initial values matter, not just the sparsity pattern. Second, the comparison is against the dense network trained for the same task, not against a smaller dense network chosen by hand. The claim is that the sparse structure found inside the large network beats a small network designed in advance.

How tickets are found: iterative magnitude pruning

The standard procedure for finding tickets is iterative magnitude pruning. It is simple, and its simplicity is part of the point: no learned pruning criterion is needed.

1. Train the dense network to completion, obtaining $\theta_{T}$ .
2. Prune a fraction of the smallest-magnitude weights, typically around one fifth per round, by setting the corresponding mask entries to zero.
3. Reset the surviving weights to their initial values $\theta_{0}$ .
4. Retrain the masked network from $m \odot \theta_{0}$ .
5. Repeat, pruning further from the retrained network, until the desired sparsity is reached.

In code-like pseudocode, one round looks like this:

# theta_0: initial weights, m: current mask (all ones at the start)
theta = train(theta_0 * m)
# prune the smallest-magnitude surviving weights
threshold = quantile(abs(theta[m == 1]), q=0.2)
m = m & (abs(theta) > threshold)
# rewind: the next round starts from the original initialisation
theta_start = theta_0 * m

One-shot pruning, where a large fraction is removed in a single step, usually performs worse than this iterative version. The iterative loop matters because each round lets the network readjust before the next cut, so the mask reflects weights that remain important after retraining, not just weights that happened to be large at the end of the first dense run.

A practical point follows immediately: this procedure is expensive. Finding the ticket requires training the dense network, then retraining at each pruning round. The ticket is therefore not, in its original form, a way to save training compute. It is a scientific claim about what dense training is doing, and a starting point for cheaper methods that try to find good sparse networks without the full iterative loop.

Why rewinding became necessary

The original formulation reset weights all the way back to

$\theta_{0}$ .

That works well for small networks and small datasets. For larger networks, resetting to the very start often fails: the sparse network trains worse than the dense network, and the gap grows with depth and task difficulty.

The fix that rescued the hypothesis at scale is rewinding. Instead of resetting to

$\theta_{0}$ ,

reset to the weights after a small number of early training steps,

$\theta_{k}$

for some small

$k$ .

In other words, train

$f(x; m \odot \theta_{k})$

rather than

$f(x; m \odot \theta_{0})$ .

Early training appears to move weights into a region where the sparse structure is stable, even though the network has not yet learned the task. After that brief warm-up, the ticket can be trained sparsely to full accuracy.

This refinement changes the interpretation slightly. The ticket is not purely a property of the random initialisation. It is a property of the initialisation plus a short, shared early trajectory. That still leaves the core surprise intact: a sparse network, identified from a dense run, can be retrained from an early point to match the dense result. But it weakens the strongest reading, in which the winning ticket is fully present before any training at all.

Rewinding also connects to a broader idea in optimisation: early training steps may determine which basin, or which broad region of parameter space, the network ends up in, while later steps refine within that region. If the mask is chosen after the network has settled into a region, and the weights are rewound only to the start of that region, the sparse retraining stays inside it. Resetting to a random point risks landing in a different region where the same mask is no longer a good structure.

What the hypothesis explains, and what it does not

The hypothesis helps explain why overparameterised networks train so well. A large network contains many overlapping subnetworks. Training a dense network can be viewed, in part, as a search over those subnetworks: gradient descent strengthens useful connections and weakens useless ones, and magnitude pruning later reveals which subnetwork won. Overparameterisation then helps because it increases the chance that a good subnetwork exists inside the initial network, not only because it gives gradient descent more directions to move in.

It also gives a clean way to think about pruning. Many pruning methods are evaluated only by post-pruning accuracy, which confounds two questions: is the sparsity pattern good, and are the remaining weights good? The ticket test separates them. A pattern is a winning ticket only if it trains well from its early weights. A pattern that only works when it inherits fully trained weights is a compressed model, not a ticket, and the distinction matters if your goal is cheaper training rather than cheaper inference.

There are limits, and they are important. First, tickets found on one task do not generally transfer to a very different task. The mask encodes structure that is useful for the task it was found on. Second, iterative magnitude pruning is a diagnostic tool, not yet a practical training algorithm, because of its cost. Methods that try to find sparse networks during a single training run, or at initialisation without any training, are motivated by the hypothesis but do not inherit its guarantees. Third, the hypothesis is clearest for image classification with standard architectures. Its status for very large language models, for reinforcement learning, and for other settings is more mixed, and claims there need task-specific evidence rather than analogy.

Common pitfalls

Treating post-training pruning as a ticket. Keeping trained weights and deleting small ones gives a small model, but it does not show that the subnetwork could have been trained from scratch. Always check the retraining condition.

Reinitialising the ticket. If you find a mask and then train it from fresh random weights, you are testing a different claim. The original effect depends on the specific early weights, so a fresh initialisation that fails does not refute the mask, and a fresh initialisation that succeeds is a different, and arguably stronger, result.

Pruning too much in one step. One-shot pruning to high sparsity often collapses accuracy even when iterative pruning to the same sparsity succeeds. The failure may be in the pruning schedule, not in the existence of a ticket.

Comparing against the wrong baseline. A sparse network should be compared with the dense network it came from, trained for a comparable number of steps, and ideally with a small dense network of similar size. Beating a weak baseline proves little.

Assuming tickets save training cost automatically. In the original procedure they do not, because finding the ticket costs several training runs. Any claim about saved compute needs a method that finds the sparse network cheaply, and a full accounting of that search cost.

A practical takeaway

If you prune models in practice, the lottery ticket literature suggests a useful discipline: separate the search for structure from the training of weights, and test whether your sparse structure can be retrained rather than only whether it survives pruning. If it can, you have something stronger than a compressed model: a structure that is genuinely easy to optimise. If it cannot, you still have a useful compressed model, but you should not claim the training savings that a true ticket would imply.

The deeper lesson is about initialisation. Random initialisation is often treated as a neutral starting point, a detail to be tuned away. The lottery ticket hypothesis suggests it is more than that: inside a random network, some subnetworks are already positioned to learn well, and training is partly the process of discovering them. Designing initialisations, architectures, and pruning methods that make good subnetworks more likely, and easier to find, remains an open and practical research direction.

---

Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial