Trained networks are substantially redundant. Pruning exploits that by removing what is not carrying weight. The catch is that removing parameters and going faster are two different achievements.
Magnitude pruning
The standard baseline: remove the weights with the smallest absolute values, on the assumption that near-zero weights contribute least. Typically done iteratively, pruning a fraction and fine-tuning to recover, rather than all at once.
It is crude and it works remarkably well, which is itself informative about how much slack trained networks carry.
The distinction that decides whether you gain anything
Unstructured pruning removes individual weights wherever they happen to be. You get a sparse matrix with zeros scattered through it. The model has fewer non-zero parameters, but the tensor is the same shape, and standard GPU kernels multiply zeros exactly as fast as they multiply anything else.
The result is a model that is smaller in a sparse storage format and no faster at all unless you have hardware and kernels built for sparsity. This catches people out constantly.
Structured pruning removes whole units: entire attention heads, channels, or layers. The tensors genuinely shrink. The model is dense, smaller, and actually faster on ordinary hardware.
Structured pruning is harder to do without hurting quality, because you are removing everything a unit did rather than the least important connections. It is also the only kind that reliably delivers a speedup.
The lottery ticket hypothesis
Frankle and Carbin proposed in 2019 that a randomly initialised dense network contains a sparse subnetwork which, trained from the original initialisation, can match the full network's performance in comparable time. They called these winning tickets.
The implication is philosophical as much as practical. It suggests overparameterised training is not wasteful so much as a search: you train a large network to find the small one that was sufficient. Reproducing the effect at very large scale has proved difficult, but it reframed how the field thinks about redundancy.
Pruning against the alternatives
- Quantization keeps every parameter at lower precision. Simpler, more reliable, usually a bigger win. Start here. See What Is Quantization?
- Pruning removes parameters entirely. Only structured pruning buys speed.
- Distillation trains a new smaller model. Most expensive, most effective, and the only one that lets you change architecture.
In practice quantization is nearly always tried first, structured pruning second, and distillation when the first two are insufficient. All three compose.
Common questions
What is model pruning?
Pruning removes weights or structural units from a trained network on the basis that they contribute little to its output. Done carefully it reduces model size, and depending on the type of pruning it may also reduce inference time.
What is the difference between structured and unstructured pruning?
Unstructured pruning removes individual weights wherever they are, producing a sparse matrix that standard hardware cannot usually exploit, so the model is smaller on disk but not faster. Structured pruning removes whole units such as attention heads or channels, producing a genuinely smaller dense model that does run faster.
What is the lottery ticket hypothesis?
The proposal, from Frankle and Carbin in 2019, that a randomly initialised network contains a small subnetwork which, trained in isolation from the original initialisation, can match the full network's accuracy. It reframes pruning as finding a subnetwork that was always sufficient rather than damaging a trained one.