Pruning removes the parts of a trained network that contribute very little.
Weights sitting near zero can often be deleted with almost no loss of quality.
Removing scattered individual weights gives a sparse model that most hardware cannot speed up.
Removing whole channels or attention heads is blunter but delivers a real speedup.
A short retrain afterwards recovers most of what was lost.
Pruning removes the parts of a trained network that contribute very little.
Trained networks are heavily over-parameterised, and many weights sit near zero.
Deleting those changes the output barely at all.
Unstructured pruning removes individual weights wherever they are.
It can remove a large fraction with almost no loss of quality.
The result is a sparse matrix that most hardware cannot compute any faster.
So the memory shrinks and the speed does not, which is a disappointing outcome.
Structured pruning removes whole units: channels, filters, layers or attention heads.
That leaves a smaller dense model, which really is faster on ordinary hardware.
It is blunter, so quality falls further for the same reduction.
Deciding what to remove is the interesting part, and magnitude is the simplest signal.
Better methods estimate each weight's contribution to the loss.
A short retrain afterwards recovers most of what was lost.
Iterative pruning alternates removing a little and retraining, and beats one large cut.
In transformers, many attention heads can be removed with surprisingly little damage.
Magnitude pruning is the baseline and it is hard to beat: sort the weights, zero the smallest, and the network usually barely notices.
Trained networks tolerate 50 to 90 percent sparsity with a point or two of accuracy loss, which says most weights contribute very little individually.
The principled criterion uses the second derivative: the loss increase from zeroing a weight depends on curvature as well as magnitude.
That is why a large weight in a flat direction can go and a small one in a sharp direction cannot.
The Hessian is intractable at scale, so magnitude survives as the practical approximation.
Unstructured pruning removes individual weights.
Zeroing scattered weights gives a sparse matrix that dense hardware multiplies at exactly the same speed as before. The model is smaller on disk and no faster.
Structured pruning removes whole channels, heads or layers.
Removing an entire output channel makes the matrix genuinely smaller, so the speedup is real on any hardware, and the accuracy cost is much higher for the same parameter count.
Semi-structured patterns are the compromise recent accelerators support directly, giving roughly a twofold speedup at a regular 50 percent sparsity.
Iterative pruning beats one shot by a wide margin.
Gradual pruning with recovery between rounds reaches far higher sparsity than a single large cut, because the network can redistribute what it lost.
There is a subnetwork that trains to the same accuracy from the original initialisation.
Pruning a trained network, then resetting the surviving weights to their original random values and retraining, can match the full network.
The same subnetwork trained from a fresh random initialisation does not, so what matters is the combination of the structure and the specific starting values.
It is a striking result about why overparameterisation helps, and it has not turned into a practical method, because finding the ticket requires training the full network first.
Quantisation makes a model 4 times smaller, reliably, in minutes.
Pruning makes it 2 times smaller, needs fine-tuning, and speeds it up only if structured.
Quantisation is almost always the first thing to try, and the two compose: a pruned and quantised model is smaller than either alone.
Distillation competes differently by training a smaller dense model rather than emptying a large one, and it usually wins when a target size is fixed in advance.
One question on this card. Every option explains itself once you pick, right or wrong.
23 concepts in this module, then a graded test with every answer explained.