Quantization Explained: How to Shrink a 70GB Model Into 4GB

01 The Precision Trade-Off

Neural network weights are typically stored in 16-bit floating point (FP16). A 70B parameter model in FP16 uses 140GB of memory. Quantization reduces this by storing weights in lower precision — 8-bit, 4-bit, or even 2-bit. Q4 (4-bit) stores a 70B model in 40GB — a 3.5x reduction. The quality loss from FP16 to Q4 is typically 1-4% on standard benchmarks. This is the key insight that enables local AI: quantization makes large models small enough to run on consumer hardware.

02 How It Works

Quantization maps continuous weight values to a smaller set of discrete values. In 4-bit quantization, weights are grouped into 16 buckets. The key technique (GPTQ, AWQ, GGUF) is to identify which weights are most important and preserve their precision while aggressively quantizing less important weights. This 'mixed precision' approach is why Q4 loses so little quality — the important weights stay near full precision while the rest are compressed.

03 Which Level Should You Use?

Q4_K_M is the sweet spot for most users: 4-bit quantization with moderate precision on key weights. It reduces a 70B model to 40GB (fits on a 48GB GPU or a 64GB Mac) with less than 2% quality loss. Q8 for production quality (no perceptible quality loss, 70GB). Q2 for extreme edge cases (25GB, but 15% quality loss — noticeable). FP16 for training and research (140GB, full precision). The quantization level is the most important decision in local AI deployment: it determines what hardware you need and what quality you get.