QuantContextAim for smooth chat (~30+ tok/s est.) · fit includes context memory
ShowHides datacenter cards (H100, etc.)
Full comparison table
Hidden by default so you can decide from the picks above. Open only if you want every model × GPU cell.
Loading data…
FAQ
How the numbers work, in plain terms. Full math lives on the methodology page.
How are tok/s numbers estimated?
Decode (generation) at batch size 1 is mostly memory-bandwidth bound, so each estimate is: tok/s ≈ (memory bandwidth GB/s × utilization) ÷ (active params in billions × bytes per param). Utilization is 35% of theoretical peak when the model fits in VRAM, 8% for CPU-only, and falls as (VRAM ÷ model size)² when it does not fit. The relative ranking between GPUs is more reliable than the absolute number.
What does "fit" mean?
A model fits when its estimated memory need — weights + KV cache at the selected context length + ~1 GB runtime overhead — stays within the card's VRAM (a negligible ~25 MB display buffer is subtracted). Fit is not weights-only: an 8 GB card can hold ~8B Q4 weights alone but still fail at 8K context, which we mark as No. Default context is 8K; you can switch 4K / 8K / 32K.
Q4 vs FP16 — which one should I look at?
Q4 (~0.55 bytes per param) is the typical local quantization: much smaller and faster per token, with a modest quality trade-off. FP16 (2.0 bytes per param) is full precision — roughly 4× the memory and bandwidth cost. The default view is Q4; switch any time in the Quant row.
Why does a measured number differ from the estimate?
Cells labeled measured are real published or curated results that override the estimate for that model × GPU × quant. Real inference engines (llama.cpp, vLLM, MLX, …), batch size, and kernels can move results by tens of percent, so treat estimates as a shopping guide, not a purchase certificate.
Is there an API?
Yes. The same JSON the site renders is available at /api/data (models, hardware, and measured anchors). Estimate math runs in the browser with the same constants; the formula is documented on the methodology page. The API is read-only and needs no key.
Are the prices live?
No. The dollar figures are MSRP (manufacturer list price), not live street prices. Buy links open a specific Amazon product page when we have an ASIN, otherwise an Amazon search — the actual price is whatever Amazon shows after you click. Links are affiliate (Amazon Associates) and may earn us a commission at no extra cost to you.
How much VRAM do I need for a 7B model?
For a typical 7B-8B model at Q4 (~4.6 GB weights) with 8K context (~1.6 GB KV cache) you need about 7–8 GB VRAM. At FP16 (~16 GB weights + KV) you'd need 18+ GB. For Qwen3-8B or Llama 3 8B at Q4, an RTX 3060 12 GB or RTX 4060 8 GB works well. At 32K context, add ~6.4 GB to the KV cache — then 12+ GB is recommended even at Q4.
Can I run LLMs on Apple Silicon Macs?
Yes. Apple Silicon Macs share system RAM as "unified memory" between CPU and GPU, which means an M4 24 GB Mac can run models that a 12 GB NVIDIA card cannot. Q4 32B models (~18 GB weights) run on an M4 Max 32 GB, Q4 70B models (~39 GB) need an M4 Max 64 GB or M2 Ultra 64 GB+. Tok/s is lower than a dedicated GPU, but 16–40 tok/s on M4 Max for a 7B Q4 model is usable for chat. See our model pages for specific Apple Silicon estimates.
What's the minimum GPU for running Llama 3 8B?
At Q4 with 8K context, Llama 3 8B needs roughly 7–8 GB VRAM. An RTX 3060 12 GB or RTX 4060 8 GB is the practical minimum for smooth chat. Even a GTX 1660 6 GB can technically fit Q4 weights but may fail at 8K context (6 GB fills up with weights alone). For FP16 at 32K context you'll want an RTX 4090 24 GB or similar. See the full compatibility matrix on our hardware pages.
Can I run LLMs without a dedicated GPU?
Yes, but much slower. CPU inference (llama.cpp with CPU backends) runs at ~5–10 tok/s for a Q4 7B model on a modern desktop CPU, which is usable for chat. A Q4 1.5B–3B model like Qwen3-3B can reach 20–40 tok/s on CPU — fine for autocomplete or summarization. Larger models (32B+) on CPU drop to 1–2 tok/s, which is impractical for interactive use. Apple Silicon's unified memory offers a middle ground without a dedicated GPU.