Every team that tries to self-host an open-weight LLM runs into the same wall within the first hour: the model does not fit. The checkpoint download finishes, the loader starts, and then the process dies with an out-of-memory error long before the first token is generated. The usual fix is a single word — quantization — but it is often applied like a magic spell, with little understanding of what it actually does to the model.
This post walks through weight quantization from first principles: why model memory is dominated by weights, what changing a number’s dtype really costs, how modern post-training quantization methods preserve accuracy far better than naive rounding, and how to decide which point on the precision ladder is right for your deployment.
The VRAM wall is an arithmetic problem
Model weights are stored as floating-point numbers, historically 16-bit half precision (FP16) or its brain-float cousin BF16. Memory consumption is therefore a trivial multiplication: parameter count times bytes per parameter. A 70-billion-parameter model at FP16 needs roughly 140 GB just for the weights — before you reserve a single byte for the KV cache, activations, or framework overhead. That is why a single 80 GB accelerator cannot serve a 70B model at half precision no matter how clever the loading code is.
The dtype ladder goes down from there. 8-bit integers halve the bill to about 70 GB, and 4-bit integers bring it to roughly 35 GB — comfortably within a single high-end GPU, with room to spare for context. Nothing about the architecture changes; the same 70 billion numbers are stored, just more coarsely. The entire discipline of quantization is about making that coarseness hurt as little as possible.
Quantization is rounding with bookkeeping
The core operation is almost embarrassingly simple. You take a group of floating-point weights, find their minimum and maximum, and map that range onto the integer grid the target bit-width allows. For 4 bits that grid has 16 values. Two bookkeeping values — a scale factor and a zero point — record how to map back from integers to floats. Storing one scale per group of, say, 64 or 128 weights costs almost nothing but lets the quantized values track the original distribution closely.
Here is a minimal asymmetric quantizer, reduced to the essentials that every production implementation shares:
def quantize_block(w, bits=4):
qmin, qmax = 0, 2**bits - 1
w_min, w_max = min(w), max(w)
scale = (w_max - w_min) / (qmax - qmin)
zero_point = round(-w_min / scale)
q = [min(max(round(x / scale) + zero_point, qmin), qmax) for x in w]
return q, scale, zero_point
def dequantize_block(q, scale, zero_point):
return [(x - zero_point) * scale for x in q]
w = [0.13, -0.72, 0.44, -0.05, 0.91, -0.33]
q, scale, zp = quantize_block(w, bits=4)
print(dequantize_block(q, scale, zp))
Run it and the reconstructed weights land close to the originals, but not exactly — each value is snapped to the nearest of 16 grid points. That per-weight rounding error is tiny in isolation. The reason naive rounding fails at model scale is that errors accumulate across billions of weights and dozens of transformer layers, and the accumulated noise is enough to visibly degrade the model’s output quality. Production quantizers add three refinements: finer groups (smaller ranges per scale), smarter rounding decisions, and in some cases a small calibration dataset to guide those decisions.
Why the clever methods work
The first refinement, popularized by GPTQ, treats quantization as an optimization problem rather than a rounding problem. Using a small sample of representative inputs, it quantizes weights one column at a time and compensates for each rounding error by adjusting the not-yet-quantized weights, using second-order information about the layer’s error surface. The result is a 4-bit model whose output distribution is dramatically closer to the original than plain rounding would produce.
The second refinement, AWQ, starts from an observation about activations: only a small fraction of weight channels are actually critical to the model’s output, and those channels are identifiable from activation magnitudes on calibration data. AWQ protects that handful of salient channels — effectively keeping them in higher effective precision — while quantizing everything else aggressively. Because the sacrifice is concentrated where the model can afford it, accuracy holds up well even at 4 bits.
Both approaches need only unlabeled calibration data, which is why they are called post-training quantization: no gradient updates, no fine-tuning run, just a measurement pass over a few hundred examples. For comparison, QLoRA takes the heavier route — quantizing a model to 4 bits and then fine-tuning adapters on top — which remains the standard trick for fine-tuning large models on a single machine. Runtime-oriented tools took a third path: the llama.cpp ecosystem defines its own family of mixed-precision block formats that assign different bit widths to different weight tensors, and Transformers integrates GPTQ, AWQ, and bitsandbytes backends behind a common loading interface, alongside the bitsandbytes library for on-the-fly 8-bit and 4-bit loading.
Why weight-only quantization is enough for inference
A fair question: if the weights are 4-bit, doesn’t the model now compute in 4-bit arithmetic and lose accuracy in every matmul? In the dominant serving configurations, no. Weight-only quantization keeps the compute in FP16 or BF16 — weights are dequantized to higher precision on the fly as they stream into the matmul. The quantization buys memory footprint, not cheaper arithmetic.
That trade is better than it sounds, because autoregressive decoding is not compute-bound — it is memory-bandwidth-bound. Generating one token requires reading essentially every weight exactly once, and the GPU spends most of its time waiting on memory rather than doing math. Shrinking the weights by 4x shrinks the bytes that must be read per token by 4x, which typically translates directly into higher tokens-per-second on memory-limited hardware. Quantization is not just what makes the model fit; it is often what makes the model fast.
Weights are not the whole memory story, though. The KV cache grows with context length and batch size, and at long contexts it can rival the weights in size. Weight quantization does nothing for it — the cache holds activations, not weights — which is why serving stacks expose separate KV cache quantization options. If you deploy long-context workloads, budget for the cache explicitly instead of assuming the quantized checkpoint is the whole memory bill.
Choosing a point on the ladder
A few field-tested heuristics cover most deployments:
- 8-bit is nearly free quality-wise and the safest default when the model already fits at 8 bits. On-the-fly loading via bitsandbytes requires no calibration run at all.
- 4-bit GPTQ or AWQ is the sweet spot for single-GPU serving of large models. Prefer AWQ when you have good calibration data matching your workload; prefer GPTQ when you want broad toolchain compatibility.
- Below 4 bits is where quality degradation stops being a rounding footnote and starts being a modeling decision. Use those formats only when you can evaluate output quality on your own tasks.
- Anything user-facing deserves an eval pass on real prompts before and after quantization. Perplexity numbers on a benchmark are a screening tool, not a substitute for checking your own workload.
One caution applies to all of these: quantization interacts with the specific model. Some families hold up at 4 bits better than others, and small models generally suffer more than large ones because they have less redundancy to absorb the noise. Treat published accuracy comparisons as a starting hypothesis, not a guarantee.
Wrapping up
Quantization looks intimidating from the outside because the terminology is dense — INT4, GPTQ, AWQ, group sizes, zero points — but the underlying idea is modest: store each weight in fewer bits, keep a per-group scale to undo the damage, and choose the rounding carefully so the accumulated error stays invisible in the model’s behavior. Start at 8-bit if it fits, drop to a calibrated 4-bit format when it does not, evaluate on your own workload, and remember that the KV cache has its own memory bill that weight quantization will never pay.