LoRA

Train a difference, not the weights

Full fine-tuning updates every parameter. LoRA freezes them all and learns a small additive correction instead.

• $W_0$: the pretrained weight matrix, $d \times k$, frozen and never touched.
• $B$: a new matrix, $d \times r$, initialised to zero.
• $A$: a new matrix, $r \times k$, initialised randomly.
• $r$: the rank, and the one number you choose. Typically 8, 16 or 64.

Because $B$ starts at zero, $BA$ starts at zero, and the model at step zero is exactly the pretrained model.

The arithmetic that makes it worth doing

For a $4096 \times 4096$ attention projection, full fine-tuning trains 16.8 million parameters.

At $r = 8$ LoRA trains $8 \times (4096 + 4096) = 65{,}536$, which is 0.39% of them.

The saving compounds: optimiser state for Adam is two extra numbers per trainable parameter, so cutting the trainable count by 250x cuts optimiser memory by 250x too.

For a 7 billion parameter model in 16-bit, full fine-tuning with Adam needs over 100 GB for weights, gradients and optimiser state.

With LoRA it needs little more than the 14 GB the frozen weights take, plus activations.

The scaling factor

• $\alpha$: a constant, usually fixed while $r$ is varied.

Dividing by $r$ means the size of the update does not change when you change the rank, so a learning rate tuned at $r = 8$ still works at $r = 64$.

Without it, raising the rank would silently raise the effective learning rate as well, and the two effects would be impossible to separate.

One training step, in order

1. Forward through the frozen $W_0$ as normal.
2. Forward the same input through $A$ then $B$, scaled by $\alpha / r$.
3. Add the two outputs together.
4. Backward: gradients flow only into $A$ and $B$.
5. The optimiser updates $A$ and $B$ and nothing else.

The bet it is making

The claim is that the useful update has low intrinsic rank: adapting a model to a new task moves it in far fewer directions than it has parameters.

That is a hard ceiling, not a tendency. Whatever $A$ and $B$ learn, the correction can never have rank above $r$.

It holds well for adapting style, format and domain, and badly for teaching genuinely new knowledge. That is the honest boundary between when LoRA is enough and when it is not.

Merging

The product can be computed once and folded into the original weights, after which inference costs exactly what the base model cost.

That is the practical difference from adapters: LoRA adds no latency at serving time, because at serving time it no longer exists as a separate thing.