Catastrophic Forgetting in LLM fine-tuning

Module 10 · Fine-Tuning · 15 concepts

In depth / 15 lines

Catastrophic Forgetting, in more detail

Catastrophic forgetting is a model losing old abilities while learning a new one.

The weights that carried general ability are the same weights being updated.

Train hard enough on a narrow task and they are overwritten.

The symptom is a model that is excellent at your task and noticeably worse at everything else.

It may lose reasoning, other languages, or the ability to hold a normal conversation.

It is easy to miss, since your evaluation only measures the task you trained for.

The main causes are too high a learning rate and too many epochs on narrow data.

One to three epochs is usually enough, and more is where damage begins.

Mixing general instruction data back into the training set reduces it substantially.

Ten to twenty per cent general data is a common and effective ratio.

Adapter methods avoid most of it by leaving the original weights frozen.

That is one of the strongest practical arguments for LoRA over full fine-tuning.

Continued pretraining on a domain corpus is especially prone to it.

Always evaluate general ability before and after, not just the target task.

The name is old, coming from research on neural networks trained on tasks in sequence.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Catastrophic Forgetting

Learning the new task moves the weights the old one needed

\boldsymbol{\theta}^{*}_{B} = \arg\min_{\boldsymbol{\theta}} \mathcal{L}_{B}(\boldsymbol{\theta})

Fine-tuning optimises the new task's loss and is indifferent to the old one, so the parameters drift to wherever $\mathcal{L}_B$ is lowest.

\mathcal{L}_{A}\big(\boldsymbol{\theta}^{*}_{B}\big) \gg \mathcal{L}_{A}\big(\boldsymbol{\theta}^{*}_{A}\big)

Nothing in the objective says the old capability should survive, and in a network where every weight is shared across every task, it usually does not.

The loss on task A can rise by an order of magnitude within a few hundred steps, which is why the word catastrophic is not rhetorical.

Not all parameters matter equally

F_{ii} = \mathbb{E}\left[ \left( \frac{\partial \log p_{\theta}(y \mid \mathbf{x})}{\partial \theta_i} \right)^{2} \right]

The diagonal of the Fisher information measures how much the old task's output depends on each parameter.

A large value means moving that weight changes the old behaviour sharply. A small value means it can move freely.

\mathcal{L} = \mathcal{L}_{B}(\boldsymbol{\theta}) + \frac{\lambda}{2} \sum_{i} F_{ii} \big(\theta_i - \theta^{*}_{A,i}\big)^{2}

Elastic weight consolidation adds a quadratic penalty per parameter, weighted by its importance, which is a spring pulling the important weights back and leaving the rest loose.

It needs one pass over the old data to estimate $F$ , and then the old data is never needed again.

The cheaper defences, in the order to try them

Lower the learning rate, since forgetting is roughly proportional to how far the weights travel.
Train for fewer epochs, because most of the damage happens after the new task has already been learned.
Mix a few percent of the original data into the new batches, which is rehearsal and is the most reliable fix.
Freeze the lower layers, where the general representations live, and train only the top.
Use an adapter, so the original weights are untouched by construction.

Rehearsal mixes in 1 to 5 percent of the old data.

A small fraction is enough, because the gradient only needs to remind the model of the old distribution rather than retrain it.

Why PEFT sidesteps most of it

\boldsymbol{\theta}_{\mathrm{base}} \text{ frozen}, \qquad \|\Delta\| \text{ small}

A LoRA adapter cannot destroy what it cannot write to, and removing the adapter restores the original model exactly.

The behaviour can still shift, since the adapter changes the output of layers the rest of the model depends on, but the base capability is recoverable rather than lost.

That also converts the problem from a training one into a routing one: keep several adapters and choose at load time, rather than trying to hold every task in one set of weights.

How to know it happened

The new task's metrics will look excellent, because they are what was optimised.

Evaluate on the old tasks, every time.

Forgetting is invisible unless it is measured, so a fine-tuning run needs a regression suite of general capabilities alongside the target metric.

A model fine-tuned on medical text that quietly lost its ability to follow formatting instructions is the ordinary case, not the unusual one.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Why does LoRA suffer less from catastrophic forgetting than full fine-tuning?

Fine-Tuning

15 concepts in this module, then a graded test with every answer explained.

Study this module →