Quantization
A model's knowledge lives in billions of numbers called weights. Storing each at full precision is heavy, so quantization rounds them to fewer bits to make the model far smaller.
Each weight is normally a 16-bit number. Quantization snaps it to a coarser grid: 8-bit, 4-bit, even 2-bit. Fewer bits means each weight takes less memory, so a 70-billion-parameter model shrinks from ~140 GB to ~35 GB or less, small enough to run on a laptop. The rounded values differ slightly from the originals, but quality usually holds up well down to 4-bit. Drag to see size drop.
grey = original · teal = full precision
16-bit → 70B model ≈ 140 GB
Variants & real-world flavors
The bit-depth ladder: FP16/BF16 is full training precision; INT8 halves memory with negligible loss; 4-bit halves it again and is the sweet spot for running big models on small hardware; below 3 bits quality degrades fast. Post-training methods quantize a finished model — GPTQ and AWQ are the standard algorithms, AWQ protecting the small fraction of weights that matter most. GGUF is the packaging format used by llama.cpp and Ollama, with quality tiers like Q4_K_M in the filename. Quantization-aware training bakes the low precision in during training itself for better results at extreme compression. Rule of thumb: a larger model at 4-bit usually beats a smaller one at full precision in the same memory.
Check yourself
The 4-bit weights are not identical to the originals. Why does the model still work almost as well?
Go deeper (free): Maarten Grootendorst — Visual guide to quantization ↗