QLoRA quantizes the frozen base model to 4-bit precision.
LoRA adapters are then trained on top of that compressed base.
This cuts memory use dramatically with little quality loss.
It puts fine-tuning a large model within reach of a single consumer GPU.
It is the standard approach for fine-tuning on limited hardware.
QLoRA combines quantisation with LoRA to fine-tune large models on modest hardware.
LoRA already avoids storing gradients for the base weights.
The frozen base still has to sit in memory in full precision.
At two bytes per parameter, a 70B model needs 140 GB just to be present.
QLoRA quantises that frozen base to four bits, cutting it to roughly 35 GB.
The base is never updated, so precision matters less than it would during full training.
LoRA adapters are trained on top in higher precision.
Gradients flow back through the quantised weights without updating them.
It introduced 4-bit NormalFloat, a format matched to how weights are actually distributed.
Double quantisation compresses the quantisation constants themselves for a further saving.
Paged optimizers spill memory to system RAM when a spike would otherwise crash the run.
Together these fit a 65B fine-tune onto a single 48 GB GPU.
The quality loss against full-precision LoRA is small enough to accept in most cases.
It is the standard approach for anyone fine-tuning without a cluster.
Training is slower per step, since weights are dequantised on the fly.
LoRA already freezes the base model, so the frozen weights are read and never updated. QLoRA's observation is that a read-only weight does not need 16 bits.
The base is stored in 4 bits, dequantised to bf16 one block at a time inside the matrix multiply, and the adapters stay in full precision because they are the thing being learned.
That is what puts a 7-billion-parameter fine-tune on a consumer card, and a 65-billion one on a single 48 GB card.
Neural weights are approximately normally distributed, so a uniform 4-bit grid wastes most of its levels on values that almost never occur.
NF4 places its 16 levels at the quantiles of a normal distribution, so each level is equally likely to be used.
Each block of 64 weights carries its own scale , since a single scale across a whole tensor would be dominated by its largest outlier.
Those block scales are themselves a real cost. Quantising them too, in blocks of 256, cuts the overhead to about 0.127 bits per weight.
It is a small saving in percentage terms and it is most of a gigabyte on a large model, which is often the difference between fitting and not.
Memory use spikes when a long sequence arrives, and a spike that exceeds the card kills a run that has been going for hours.
Paged optimiser states move to CPU memory when the GPU is under pressure and come back when it is not, using the same unified memory mechanism as operating system paging.
It makes the failure mode slow rather than fatal, which is the difference between a finished run and a lost one.
Step five has a trap. Merging bf16 adapters into a 4-bit base and saving the result loses precision twice, so the merge should be done against the original weights.
Dequantisation happens on every forward pass, so training is roughly a third slower than unquantised LoRA on hardware that had the memory anyway.
Quality, on the published comparisons, is within noise of 16-bit LoRA, which is the result that made the method standard rather than a compromise.
The base model's own accuracy does drop slightly under 4-bit quantisation, and the adapter is trained against the quantised base, so it learns to compensate for exactly that error.
One question on this card. Every option explains itself once you pick, right or wrong.
15 concepts in this module, then a graded test with every answer explained.