Quantization stores weights at lower numeric precision.
Going from 16-bit to 8-bit or 4-bit halves or quarters memory use.
Post-training quantization converts an already trained model.
Quantization-aware training bakes the constraint in during training for better quality.
It is what lets large models run on a single GPU or on consumer hardware.
Quantization stores model weights at lower numeric precision.
Training typically uses 16 or 32 bits per number.
Serving rarely needs that much, since the weights are fixed and the task is forgiving.
Dropping to 8 bits halves the memory, and 4 bits quarters it.
A 70B model at 16 bits needs 140 GB, and at 4 bits about 35 GB.
That is the difference between a cluster and a single serious GPU.
It speeds things up too, since decoding is limited by reading weights from memory.
Fewer bytes to read means more tokens per second.
Post-training quantization converts an already trained model, which takes minutes.
GPTQ and AWQ use a small calibration set to choose the rounding carefully.
Quantization-aware training bakes the constraint in during training for better quality.
It costs a full training run, so it is used when the last point of accuracy matters.
Quality loss at 8 bits is usually negligible and at 4 bits is small but real.
Below 4 bits degradation becomes obvious, especially on reasoning tasks.
It is what puts capable models on consumer hardware.
The whole method is choosing and well, because the rounding error is bounded by half a step and the step is set by the range.
Decoding reads every weight to produce one token, so it is bound by memory rather than arithmetic.
Halving the bytes nearly halves the time, and the arithmetic saving is a secondary benefit that only appears if the hardware has integer units to use.
Prefill is compute-bound, so it gains less.
That is why quantisation helps generation far more than it helps processing a long prompt.
Symmetric quantisation is cheaper because the zero point disappears from the arithmetic, and it wastes half the range on a distribution that is not centred.
A single scale for a whole tensor is ruined by one outlier. A scale per output channel, or per group of 64 or 128 weights, keeps the resolution where the values are.
Finer granularity costs a little storage for the scales, and it is the difference between 4-bit quantisation working and not.
That is post-training quantisation, and at 8 bits it typically costs under a point of accuracy.
Quantisation-aware training inserts fake quantisation into the forward pass so the weights adapt to it, using a straight-through estimator for the gradient since rounding has none.
It recovers most of the loss at 4 bits and below, and it costs a full training run.
In large language models a few channels carry values dozens of times larger than the rest, and those channels alone destroy a per-tensor activation scale.
The practical methods all handle this explicitly: separate the outlier channels, or shift the difficulty from activations into weights before quantising.
GPTQ, AWQ and NF4 are the common methods.
Weight-only quantisation is the common case, since activations are produced fresh per token and weights are read constantly.
Compressing the weights buys the bandwidth and leaves the harder distribution alone.
One question on this card. Every option explains itself once you pick, right or wrong.
23 concepts in this module, then a graded test with every answer explained.