Serving a 7B model in fp16 takes roughly 14 GB of VRAM. A 70B model takes around 140 GB — which means four 40GB accelerators before you have written a single line of inference code. For most teams, quantization is what makes self-hosting a large language model economically viable at all: compress the weights to 4 bits and that same 70B model fits on a single 80GB GPU, with quality that is often indistinguishable in practice.
But “quantize the model” hides a surprising amount of variation. GPTQ, AWQ, GGUF, bitsandbytes, and SmoothQuant all reduce precision, yet they make different trade-offs in accuracy, inference speed, and hardware support. Picking the wrong one gets you either a model that runs slower than the fp16 baseline on your hardware or one that quietly degrades on the tasks you care about. This post walks through how the major methods actually work, where they shine, and how to choose between them.
What Quantization Actually Changes
Model weights are stored by default in 16-bit floating point (fp16 or bf16), so every parameter occupies two bytes. Quantization converts those weights to lower-precision representations — 8-bit integers, 4-bit integers, or exotic formats in between. The core challenge is that an LLM’s activations contain extreme outlier values, and naive rounding of weights near those outliers causes disproportionate damage to output quality.
Every modern method exists to answer the same question differently: which weights can afford to lose precision, and which ones cannot?
There is a useful distinction to keep in mind. Post-training quantization (PTQ) takes finished weights and compresses them with a calibration pass — no training infrastructure needed, which is why it dominates the open-source ecosystem. Quantization-aware training (QAT) bakes quantization into training itself and preserves more accuracy at 3 bits and below, but it costs a full training run. Everything below is PTQ, because that is what you will realistically use.
GPTQ: Curvature-Guided Rounding
GPTQ was the first method to push LLMs down to 4 bits while keeping usable accuracy. Its insight is that simple per-weight rounding is the wrong objective: instead of minimizing the error on each weight, GPTQ minimizes the change in the layer’s output caused by quantization. It uses approximate second-order (Hessian) information to decide the rounding order and compensate each quantization error with adjustments to the remaining weights.
GPTQ operates in W4A16 mode — weights at 4 bits, activations still in fp16. It quantizes layer by layer in a single pass over a calibration dataset, which makes it fast to run but means it needs a GPU with enough memory to hold the model during the process (roughly 16GB for a 7B model). New model architectures tend to get GPTQ support early through toolkits like GPTQModel — the maintained successor to the now-archived AutoGPTQ — and the format is broadly supported across inference engines.
AWQ: Protect the 1% That Matters
AWQ (Activation-Aware Weight Quantization) starts from an observation that sounds almost too good: protecting only about 1% of weights — the ones paired with the largest activation magnitudes — removes most of the quantization error. Instead of keeping those weights in higher precision (which would create a messy mixed-precision format), AWQ applies per-channel scaling before quantization. The forward pass divides activations by the same factors, so the mathematical output is unchanged, but the salient channels land in friendlier regions of the quantization grid.
The payoff shows up in two places. AWQ tends to hold up better than GPTQ on instruction-following and coding evaluations, and its dequantization kernels are faster at inference time. Published benchmarks comparing the two on a 7B model with an A100-class GPU show AWQ completing single-request inference meaningfully faster than GPTQ — a gap large enough to matter for latency-sensitive serving. Both produce W4A16 checkpoints, so switching between them is mostly a matter of re-quantizing and swapping the artifact.
One caveat: AWQ’s calibration requires a small, representative text sample. Quantizing a coding model with a calibration set of casual conversation text measurably hurts downstream performance — this is the single most common way teams get a “bad” AWQ checkpoint from a good base model.
GGUF: The llama.cpp Ecosystem
A common misconception: GGUF is not a quantization algorithm. It is a llama.cpp file format that bundles weights, tokenizer, and metadata into one portable file. The quantization methods inside it are llama.cpp’s own — a family of block-wise schemes that quantize weights in small groups with shared scale factors.
The naming encodes the trade-off space. Legacy types (Q4_0, Q4_1) are simple and fast but noticeably lossy. The k-quants (Q4_K_M, Q5_K_M, Q6_K) mix block types within a tensor — more important tensors get finer granularity — and remain the pragmatic default. The newer i-quants (IQ3_S, IQ4_NL) squeeze out better accuracy per bit, especially at aggressive sizes, at the cost of slower quantization and slightly slower inference on some hardware.
As a rough map: Q4_K_M lands near 4.8 bits per weight and is the standard “small but sane” choice; Q5_K_M at about 5.7 bpw is near-lossless for most purposes; Q8_0 at 8.5 bpw is effectively indistinguishable from fp16. The killer feature of the ecosystem is that quantization happens on CPU after a one-time conversion to GGUF — no GPU needed at all. The llama-quantize tool is a single command:
./llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_MYou can push quality further with an importance matrix — run llama-imatrix over a calibration text first, then pass it to llama-quantize so the quantizer knows which weights the model actually leans on:
./llama-imatrix -m model-f16.gguf -f calibration.txt --chunk 512 -o model.imatrix
./llama-quantize --imatrix model.imatrix model-f16.gguf model-q4_k_m.gguf Q4_K_MThis is the same activation-aware idea behind AWQ, arrived at independently — and it matters more the lower you go. At 3 bits, an imatrix is usually the difference between a usable model and gibberish.
SmoothQuant and the W8A8 Path
GPTQ and AWQ leave activations in fp16, which means inference still runs the expensive fp16 matmul kernels. SmoothQuant takes a different route: it migrates the quantization difficulty from activations into weights via a per-channel scaling transformation, enabling full W8A8 — both weights and activations in int8. On hardware with strong int8 throughput, this can beat W4A16 for batch-heavy serving even though it uses more memory per weight.
There is also an 8-bit weight-only option worth knowing: bitsandbytes’ int8 mode, which quantizes on load with zero calibration and zero preprocessing. It is the most convenient option in the ecosystem — trivially available through the Transformers integration — and the least accurate of the group. Good for experimentation, not for production latency targets.
Running These in Practice
In vLLM, quantized checkpoints are loaded with the --quantization flag. The engine detects the format from the checkpoint config, and for supported GPU generations it dispatches to fused dequant-matmul kernels (such as the Marlin kernels) that make 4-bit inference dramatically faster than naive dequantize-then-multiply:
vllm serve Qwen/Qwen2.5-32B-Instruct-AWQ --quantization awq_marlinThe hardware support matrix matters here — Marlin-class kernels target Ampere and newer NVIDIA GPUs, so an older card may fall back to slower paths. Check the vLLM quantization docs for your specific GPU generation before committing to a format.
For bitsandbytes, quantization happens transparently at load time:
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B-Instruct",
quantization_config=bnb_config,
device_map="auto",
)The NF4 (4-bit NormalFloat) type here is information-theoretically optimal for normally distributed weights, which is why bitsandbytes at 4 bits performs better than its “no calibration” story would suggest.
The Memory Math
Worth internalizing, because it drives deployment decisions more than any benchmark table:
- A 7B model: ~14 GB in fp16, ~7 GB at 8-bit, ~4 GB at 4-bit
- A 32B model: ~64 GB in fp16, ~20 GB at 4-bit — fits on a single 40GB card
- A 70B model: ~140 GB in fp16, ~40 GB at 4-bit — fits on a single 80GB card
These are weights-only numbers. Real deployments add KV cache, activation buffers, and framework overhead — budget 20-30% above the weight footprint, more if you need large batch sizes or long contexts.
Choosing a Method
The decision is mostly about your serving stack, not the algorithms:
- vLLM or TGI on modern NVIDIA GPUs: AWQ 4-bit is the reliable default. GPTQ is an equally supported fallback and sometimes lands first for brand-new architectures.
- CPU or Apple Silicon inference: GGUF. Q4_K_M for the size/quality sweet spot, Q5_K_M when quality matters more than headroom, and always generate with an importance matrix below 5 bits.
- Batch-heavy serving with int8-capable hardware: consider W8A8 via SmoothQuant-style pipelines — the memory cost is higher but integer matmul throughput can win.
- Quick experimentation: bitsandbytes on load. Zero setup, lower ceiling.
Two habits prevent most quantization regrets. First, always evaluate the quantized checkpoint on your workload before switching traffic — aggregate benchmark deltas hide task-specific regressions that show up exactly where your users are. Second, keep the fp16 weights around. Quantization formats and kernels improve every few months, and re-quantizing from source beats inheriting the noise of a previous generation’s format.
Wrapping Up
Quantization has matured from a research curiosity into routine infrastructure. GPTQ proved 4-bit was possible, AWQ showed that protecting a small fraction of weights beats uniform treatment, and the llama.cpp ecosystem made CPU-side quantization a one-command operation. None of these are exotic anymore — the practical question is just which artifact your serving stack runs fastest, and whether your calibration data looks like your traffic. Start from AWQ on GPU or Q4_K_M on CPU, measure on your own tasks, and treat the 4-bit checkpoint as the new baseline rather than a compromise.