Of all the ways to make a model cheaper to run, quantization gives the largest return for the least effort. It requires no training run, no teacher, and no dataset beyond a small calibration sample. It is usually the first thing to try.
The basic idea
Models are trained using floating point numbers, commonly 16-bit, so each parameter occupies two bytes. Quantization stores those same parameters using fewer bits, typically 8-bit or 4-bit integers, by mapping the range of values onto a coarser grid.
The architecture is unchanged. The parameter count is unchanged. Only the precision of each number goes down.
Why it matters so much
The arithmetic is stark. A 70 billion parameter model at 16-bit precision needs roughly 140 GB just to hold its weights, before any activation or context memory. That means multiple high-end accelerators.
The same model at 4-bit needs roughly 35 GB, which fits on a single card. That is not an incremental improvement. It is the difference between a multi-GPU deployment and a single-GPU one, and often between feasible and not.
Inference is usually memory-bandwidth bound rather than compute bound, so moving fewer bytes also makes generation faster, not merely smaller.
Where quality goes
Roughly, and depending on method and model:
- 8-bit: quality difference typically negligible. Close to free.
- 4-bit: small measurable degradation with a good method. Usually a clear win.
- 3-bit and below: degradation becomes pronounced and uneven, hitting reasoning and long-context behaviour hardest.
The failure mode worth knowing is outliers. A small number of activations in large transformers take values far outside the typical range, and naive schemes that fit one scale across a whole tensor waste nearly all their precision accommodating them. Most of the sophistication in modern methods is about handling this.
Post-training quantization vs quantization-aware training
Post-training quantization takes a finished model and converts it, using a few hundred calibration samples to determine scaling. It takes minutes to hours and requires no training infrastructure. This is what almost everyone uses.
Quantization-aware training simulates the precision loss during training so the model learns weights that survive it. It produces better results at aggressive bit widths, but requires a full training run, which for a large language model removes most of the reason you wanted quantization in the first place.
GPTQ
GPTQ quantises the model one layer at a time. Within each layer it processes weights in sequence, and after quantising each one it adjusts the remaining unquantised weights to compensate for the error just introduced, using approximate second-order information to decide how.
The effect is that error does not simply accumulate: each step is partially corrected by the steps that follow. It is a one-shot method needing only a small calibration set, and it is a large part of why 4-bit quantization of very large models became practical.
AWQ
AWQ starts from a different observation: not all weights matter equally. A small fraction, on the order of one percent, disproportionately affects output quality, and which ones they are can be identified from activation magnitudes rather than from the weight values themselves.
Having identified the salient weights, AWQ scales things so those retain more effective precision while the rest absorb the loss. It avoids backpropagation entirely, tends to be fast, and often generalises better to domains outside the calibration set.
In practice both are strong. The choice is usually decided by which is better supported by your serving stack.
How it relates to distillation
They solve the same problem by opposite means, and compose cleanly.
Quantization keeps the model and stores it more efficiently, retraining nothing. Distillation discards the model and trains a smaller one to behave like it, which requires a large volume of teacher output and a full training run. See What Is Knowledge Distillation?
Because they are independent, the usual production answer is both: distil to the size you need, then quantise the result. And because quantization is so much cheaper to attempt, it is worth exhausting before considering distillation at all.
A practical order of operations
- Quantise to 8-bit. Measure. This is nearly always safe.
- Try 4-bit with GPTQ or AWQ. Measure on your own evaluations, not published benchmarks.
- If it still does not fit or is still too slow, consider a smaller model, or distillation.
- Whatever you end up with, quantise that too.
Common questions
What is quantization in machine learning?
Quantization stores a model's numbers at lower precision than it was trained in, typically moving from 16-bit floating point to 8-bit or 4-bit integers. The model keeps the same architecture and parameter count, but each parameter occupies less memory and arithmetic runs faster.
What is the difference between GPTQ and AWQ?
Both are post-training methods for 4-bit weights. GPTQ quantises layer by layer using approximate second-order information to compensate for the error introduced at each step. AWQ instead identifies the small fraction of weights that matter most, based on activation magnitudes, and protects those.
Does quantization make a model worse?
At 8-bit the quality difference is usually negligible. At 4-bit with a good method it is small and often acceptable. Below 4-bit degradation becomes pronounced. The tradeoff is almost always favourable given that halving precision roughly halves memory.
Is quantization the same as distillation?
No. Quantization keeps the same model and stores it more compactly, requiring no training. Distillation trains a new, smaller model to imitate the original, requiring a full training run. They compose: a distilled model can then be quantised.