LLM Quantization Explained: Q4 vs FP16 vs Q8 vs Q2
Quantization is the single biggest lever you have when a model doesn't fit your GPU. The same 70B model can demand 156 GB of memory or 54 GB depending on one setting. This guide explains what the numbers mean, what you give up at each level, and why AI Hashrate's estimates focus on exactly two tiers: Q4 and FP16.
What is quantization, in one minute?
An LLM is a giant table of numbers called weights. In their original training format each weight is usually stored as a 16-bit floating point number — 2 bytes per parameter. Quantization re-stores those weights with fewer bits: 8 bits, 4 bits, even 2 bits per parameter, using clever scaling so the model's behavior changes as little as possible.
Fewer bits per weight means three things: less VRAM to hold the model, faster decoding (every generated token requires reading all active weights from memory, so smaller weights move faster across the memory bus), and some quality loss that grows as you go lower.
What do FP16, Q8, Q4 and Q2 actually mean?
| Tier | Bits/weight | ~Bytes/param | Typical quality impact |
|---|---|---|---|
| FP16 / BF16 | 16 | 2.0 | Reference quality |
| Q8 | 8 | ~1.0 | Nearly indistinguishable for most uses |
| Q4 | 4 | ~0.55 (with scales/metadata) | Small but sometimes noticeable loss |
| Q2 | 2 | ~0.3 | Clearly degraded; last-resort tier |
The bytes-per-parameter figures for Q4 and Q2 are approximations: real formats (GGUF, AWQ, GPTQ and friends) store extra scaling factors alongside the quantized weights, which is why a "4-bit" model averages ~0.55 bytes per parameter rather than exactly 0.5.
How much VRAM does each tier actually save?
Using the AI Hashrate formula (weights + KV at 8K context + 1 GB overhead) on a Llama-3.1-8B-class model:
| Tier | Weights | Estimated total | Example GPU that fits |
|---|---|---|---|
| FP16 | 16.0 GB | 18.6 GB | RTX 4090 24GB |
| Q8 (~1.0 B/param) | ~8.0 GB | ~10.6 GB | 12 GB card, tight |
| Q4 | 4.4 GB | 7.0 GB | RTX 3060 12GB, any 8 GB card |
| Q2 (~0.3 B/param) | ~2.4 GB | ~5.0 GB | 6 GB card |
Going from FP16 to Q4 cuts the memory bill by roughly 60% and makes decode roughly 3–4× faster on the same card, because decode speed is bound by memory bandwidth: fewer bytes per token means more tokens per second. You can see this directly in the estimated tok/s columns on any AI Hashrate hardware page.
What quality do you actually lose at Q4?
Less than most people expect, especially on larger models. The community consensus from public evaluations is that Q4-quantized versions of strong models retain the large majority of their benchmark scores, and bigger models tolerate 4-bit quantization better than small ones — a 70B at Q4 is generally considered more capable than a 13B at FP16, while using similar memory. Where Q4 loss shows up most: long multi-step reasoning chains, precise arithmetic, and rare-knowledge recall. For chat, summarization and coding assistance, most users find Q4 of a strong model perfectly serviceable.
We deliberately make no numerical quality claims here — quality is benchmark-dependent, and this site's numbers are memory and speed estimates only.
When is Q2 worth it?
Almost never as a first choice. Q2 exists for one scenario: running a much larger model than your hardware should allow, accepting visible degradation — for example a 70B at Q2 squeezed onto a 24 GB card. Even then, an MoE model with small active params or a mid-size Q4 model usually beats it in practice. Treat Q2 as an experiment, not a daily driver.
Why does AI Hashrate only estimate Q4 and FP16?
Three reasons:
- They bracket the real decision. FP16 is the quality ceiling; Q4 is the practical sweet spot most local users actually run. If you know a model fits at FP16 and flies at Q4, the middle tiers are interpolation.
- Honest math over false precision. Q8 formats vary (~1.0–1.06 B/param) and Q2 formats vary wildly. Publishing per-tier estimates for formats we can't model cleanly would invent precision we don't have. Q4 (~0.55) and FP16 (exactly 2.0) are stable conventions.
- Measured anchors map to these tiers. Where we hold measured tok/s, real-world submissions overwhelmingly report Q4-class (GGUF Q4, AWQ, INT4) or FP16-class (FP16, BF16) runs — our methodology explains how measured rows override estimates.
So which quant should you pick?
Simple decision rule: run the largest model that fits your VRAM at Q4 with room to spare; only step up to FP16 when the model fits comfortably and you have a quality-critical task; only step down below Q4 when nothing else fits. Check your exact GPU against any model on the AI Hashrate home page — the fit/speed matrix computes both tiers at 8K context instantly.