How Much VRAM Do You Need for Local LLMs? (3B to 100B+)

Estimated requirements, Q4 and FP16, default 8K context · AI Hashrate guide

The honest answer is: more than the model's file size. Running a local LLM takes weights + KV cache for your context + runtime overhead. On AI Hashrate we estimate all three with one formula, so this guide uses the same numbers you will see on every GPU and model page. Everything below assumes our default 8K context and the rule that a model only counts as "fitting" when it needs ≤ 95% of your VRAM.

How are these VRAM numbers calculated?

Our estimate has three parts:

Fit = weights + KV cache + 1 GB overhead, where weights = total parameters × bytes per parameter (0.55 for Q4, 2.0 for FP16) and KV cache ≈ active parameters × (context ÷ 1024) × 0.025 GB. MoE models store all experts (total params) but only pay KV for active params. You can read the full reasoning on our methodology page.

How much VRAM does a 3B model need?

Small models like Llama-3.2-3B (3.2B params) are the easiest tier:

Any GPU with 4 GB+ runs the Q4 comfortably, and a 12 GB card like the RTX 3060 12GB even fits FP16. This tier is ideal for older cards and laptops.

How much VRAM for 7B–8B models?

The most popular local tier (Llama-3.1-8B, Qwen3-8B, Mistral-7B):

Q4 fits on an 8 GB card (7.6 GB usable at our 95% rule), but FP16 wants a 24 GB card such as the RTX 4090 or RX 7900 XTX. This is why we call 8B-at-Q4 the sweet spot of local inference: it runs almost everywhere.

How much VRAM for 13B models?

Q4 fits on 12 GB cards (11.4 GB usable); FP16 needs roughly a 32 GB card like the RTX 5090 32GB. Note how close the 12 GB Q4 case is to the limit — longer context will push it over.

How much VRAM for 30B-class models?

Using Qwen3-32B (32.8B params) as the reference:

This is the tier that surprises people: a 24 GB RTX 4090 (22.8 GB usable) does not fit a 32B Q4 at 8K context, while the 32 GB RTX 5090 (30.4 GB usable) does. FP16 requires an 80 GB datacenter card like the A100 80GB. One exception: MoE models in this size class, like Qwen3-30B-A3B, have much smaller KV caches because only ~3B params are active.

How much VRAM for 70B models?

Llama-3.1-70B (70.6B params):

Q4 needs an 80 GB card (76 GB usable) — a 48 GB card like the RTX 6000 Ada falls short at 8K. FP16 needs a 192 GB accelerator such as the MI300X, or a unified-memory machine like the 128 GB MacBook Pro M4 Max for Q4.

How much VRAM for 100B+ models?

Two very different stories here:

No single consumer GPU runs a dense 405B at any quant we track. MoE flagships are far more reachable: Qwen3-235B Q4 fits on a 192 GB MI300X or a 192 GB Mac Studio M2 Ultra, because KV cache scales with the 22B active params, not the 235B total.

Does context length really change the answer?

Yes — KV cache grows linearly with context. Doubling from 8K to 16K doubles the KV column. For a dense 70B at FP16 that means another 14 GB; for an 8B model, only 1.6 GB. If a model barely misses the fit line at 8K, halving context to 4K is often enough to make it fit.

What's the takeaway?

As a rule of thumb at 8K context: 8 GB runs 8B Q4, 12 GB runs 13B Q4, 32 GB runs 32B Q4, 80 GB runs 70B Q4. FP16 roughly triples the requirement. For the per-GPU, per-model answer — including estimated tokens/second — use the interactive matrix on the AI Hashrate home page.