Full Fine-Tuning vs PEFT, explained in 5 lines

Module 10 · Fine-Tuning · 15 concepts

In depth / 15 lines

Full Fine-Tuning vs PEFT, in more detail

Both approaches adapt a pretrained model to your task, and they differ in how much they change.

Full fine-tuning updates every weight in the network.

Parameter-efficient fine-tuning freezes the original weights and trains a small addition.

The size difference is dramatic in practice.

A seven billion parameter model tuned fully means updating and storing all seven billion.

A LoRA adapter for that same model can be a few tens of megabytes.

Training memory is the binding constraint, because optimiser state is several times model size.

Full fine-tuning of a mid-sized model needs several high-memory GPUs.

PEFT often fits on a single consumer card, which is the reason it became the default.

Deployment differs just as much as training does.

Full fine-tuning leaves you a separate complete model for every task you support.

PEFT leaves you one shared base model and a small swappable file per task.

Quality is close for most adaptation, and full fine-tuning keeps an edge on large shifts.

Teaching an unfamiliar domain or a new language is where the extra capacity earns its cost.

Start with PEFT, and move to full fine-tuning only once you can show that it is needed.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Full Fine-Tuning vs PEFT

The memory bill of a full fine-tune

M = M_{\mathrm{weights}} + M_{\mathrm{grads}} + M_{\mathrm{optimiser}} + M_{\mathrm{activations}}

Weights in bf16 cost 2 bytes each. Gradients cost another 2. Adam keeps two moments in fp32, which is 8 more, plus an fp32 master copy at 4.

2 + 2 + 8 + 4 = 16 \text{ bytes per parameter}

7 \times 10^{9} \times 16 = 112 \text{ GB}

A 7-billion-parameter model needs about 112 GB before a single activation is stored, which is more than any single accelerator has.

The weights themselves are only an eighth of that. The optimiser state is the problem, and it is proportional to how many parameters are being trained.

What PEFT changes

\boldsymbol{\theta} = \boldsymbol{\theta}_{\mathrm{frozen}} + \Delta\boldsymbol{\theta}, \qquad |\Delta\boldsymbol{\theta}| \ll |\boldsymbol{\theta}|

Freeze the model and train a small set of new parameters instead. Gradients and optimiser state are then needed only for that set.

M \approx 2|\boldsymbol{\theta}| + 16|\Delta\boldsymbol{\theta}|

14 + 0.3 = 14.3 \text{ GB}

With 0.1 percent of the parameters trainable, the same model fits on one 24 GB card, and the arithmetic per step barely changes.

The forward pass is unchanged, so the saving is entirely in what has to be differentiated and remembered.

The four families

Additive: insert small trainable modules, as adapters do.
Reparameterised: learn a low-rank update to existing weights, as LoRA does.
Selective: unfreeze a chosen subset, such as the biases or the last few layers.
Prompt-based: prepend trainable vectors to the input and change no weights at all.

\text{LoRA} : W + BA, \quad B \in \mathbb{R}^{d \times r}, \; A \in \mathbb{R}^{r \times k}

Reparameterised methods dominate in practice because they add nothing at inference: the update is merged back into the weight matrix and the model runs at its original speed.

What full fine-tuning still wins

A new capability is not the same thing as a new behaviour.

Teaching a model a format, a tone or a task it can already almost do is a behaviour change, and PEFT matches full fine-tuning on those.

Teaching it a genuinely new domain, a new language or knowledge it does not have needs capacity that a low-rank update does not provide.

As a rule of thumb, full fine-tuning pays off above about 100,000 examples and a real domain shift.

Below that, full fine-tuning usually costs twenty times as much for a difference inside the noise of the evaluation.

One decision, in order

Check whether prompting or retrieval already solves it, since neither trains anything.
If training is needed, start with LoRA at a small rank.
Raise the rank before considering a full fine-tune, since that is the cheaper axis.
Move to full fine-tuning only when more data and more rank have both stopped helping.
Keep the base model and the adapter separate, because one base can serve many adapters.

That last point is the operational argument. A full fine-tune produces a new 14 GB model per task, and an adapter produces a 20 MB file that loads onto the model already in memory.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What is the main practical advantage of PEFT over full fine-tuning?

Fine-Tuning

15 concepts in this module, then a graded test with every answer explained.

Study this module →