An adapter is a small trainable module inserted between frozen layers.
Only the adapter's parameters are updated during training.
Different adapters can be trained for different tasks on one shared base.
You load whichever adapter the current request needs.
It is much cheaper than storing a full model copy per task.
An adapter is a small trainable module inserted into a frozen model.
The original weights never change, and only the adapter learns.
The classic design places a small bottleneck block inside each transformer layer.
It projects down to a small size, applies a non-linearity, and projects back up.
It starts near identity so the untrained model behaves exactly as before.
That means training begins from the base model's behaviour rather than from noise.
Adapters typically add one to five per cent of the parameters.
Each task gets its own adapter and they all share one base model in memory.
You load whichever adapter the request needs, which takes milliseconds.
Serving fifty customer-specific variants becomes one model plus fifty small files.
The classic adapter sits in the forward path and adds a little inference latency.
LoRA avoids that by being mergeable into the base weights.
That is largely why LoRA became the default choice.
Adapters can be composed or stacked, though results are inconsistent.
The general idea, small modules over a frozen base, is now standard practice.
Two small matrices and a nonlinearity are inserted after each sublayer. Everything around them is frozen.
With and that is 131,000 parameters per insertion point, so a 32-layer model with two per layer trains about 8 million of them.
The up projection is initialised to zero, so the adapter contributes nothing at step one and the model behaves exactly as it did before.
Training then moves it away from zero gradually. Without that, inserting random matrices into a trained network destroys its behaviour before any learning happens.
The residual connection around the bottleneck is what makes the identity reachable at all, and it is the same reasoning as in a transformer block.
Leaving the layer norm parameters trainable is a small addition that reliably helps, since it lets the model rescale what the adapter changed.
Latency rises by 2 to 6 percent.
An adapter is a genuine extra operation in the forward pass, so it is still there at inference and every token pays for it.
LoRA's update is a linear term parallel to an existing weight matrix, so it can be added into that matrix once and disappear entirely.
That single property is why LoRA displaced adapters as the default, despite the two performing about the same on quality.
LoRA and adapters both serve many tasks from one base model, and adapters do it without the merge step, so switching task is swapping a small module rather than rewriting weights.
AdapterFusion is the clearest example.
Several adapters trained on different tasks can be combined with a learned attention over their outputs, which is a composition story LoRA has no direct equivalent for.
The nonlinearity inside the bottleneck also gives them strictly more expressive power than a low-rank linear update at the same parameter count, which occasionally shows on hard tasks.
One question on this card. Every option explains itself once you pick, right or wrong.
15 concepts in this module, then a graded test with every answer explained.