2026 Local LLM GPU Buying Guide: What Actually Runs What

July 2026 · Data from aihashrate.stream — 73 models, 63 GPUs, 100+ measured anchors

Short version: For smooth Q4 7–9B chat, get 12GB+. For 30B+ dense models, 24GB+ with offload. For MoE models, VRAM budget for total params, not just active params. The RTX 4090 is still the best consumer card you can just buy on Amazon right now. If you're on a budget, a used RTX 3060 12GB punches way above its weight.

This is not a lab leaderboard. These numbers come from memory-bandwidth estimation calibrated against 100+ community-measured tok/s anchors. They're directionally right — but your specific backend, quant engine, and batch size will move the needle. Use this to narrow your options, then check the interactive tool with your exact model and context length.

1. VRAM Classes: What Actually Fits (Q4, 8K Context)

Every few days someone on r/LocalLLaMA asks: "Can I run X on Y VRAM?" Here's the real answer. These numbers use the standard formula: weights + KV cache + 1GB overhead. Q4 quantization, 8K context. For longer context, add more KV budget. For FP16, roughly double the weight budget.

If a model lands in "borderline", it means the weights fit but the KV cache + overhead pushes total over — you might need to drop context or offload layers. Models are ranked by HuggingFace download popularity.

Want to check a specific combo? Pick GPU + model on ai-hashrate and it'll show exact VRAM breakdown, tok/s estimate, and whether it's measured or estimated.

2. Measured Tok/s Anchors: Real Numbers, Not Marketing

These are real measurements from community testers, not estimates. Grouped by GPU. Each GPU runs different models, so the range reflects both model size and quant variation. Higher = better, but "usable for chat" starts around 20 tok/s. Smooth chat needs 30+.

3. The MoE VRAM Trap — Everyone Gets This Wrong

Mixture-of-Experts models advertise small "active parameters" — Qwen3.6-35B-A3B activates only 3B per token. Sounds like it should fit on a potato. Doesn't work that way.

The rule: VRAM must hold ALL weights, not just active experts. Speed scales with active params. But if you don't have VRAM for all 35B weights, the model won't load — period.

Bottom line: When someone says "this 35B MoE only activates 3B, so it's basically a 3B model" — they're wrong about VRAM. It activates like a 3B model (fast), but it consumes VRAM like a 35B model. The website at ai-hashrate handles this automatically — MoE models show both active and total params in the estimate.

4. Tokens Per Dollar: Where's the Value?

If you just want the most tok/s for your money — regardless of VRAM — here's the ranking. Uses Q4/8K estimate for a typical 8B model, divided by MSRP.

Value rankings use MSRP. Used market prices shift everything — an RTX 3090 at $700 used beats almost everything new. But we can't track used prices in real time, so we use MSRP as the anchor.

5. Quick Picks: What to Buy at Each Budget

6. What About Apple Silicon?

Apple's unified memory is a different beast. An M4 Mac mini with 32GB has only 120 GB/s bandwidth — but it can run models that need 20+ GB of VRAM because the memory is shared. At Q4, it fits everything up to gemma-4-31B and Qwen3.6-35B-A3B MoE.

The trade-off: speed. You're getting 20–40 tok/s instead of 100+. For interactive chat, 30 tok/s is fine. For batch processing or speculative decoding, NVIDIA wins by a mile.

Measured anchors: Mac mini M4 32GB does 31 tok/s avg on Q4 models. M2 Ultra gives you more bandwidth (800 GB/s) but costs $5,000+ — at which point you could buy two RTX 4090s.

Where These Numbers Come From

Every estimate on this page uses the same formula as our methodology:

• VRAM = weights + KV cache + 1GB overhead. Weights are params × bytes-per-param (Q4 ≈ 0.55 bytes, FP16 ≈ 2 bytes). KV cache scales with context length.
• Tok/s ≈ bandwidth / bytes-per-token × utilization factor. We use 0.35 for fitting models (conservative), lower for offloaded models. The formula is simple physics — memory bandwidth is the bottleneck, not compute.
• Measured anchors calibrate the estimates. When community testers report real tok/s numbers, we record them and use estimates for the rest. Look for "measured" vs "estimated" tags on the site.

These are ballpark numbers, not lab benchmarks. Your actual tok/s depends on your backend (llama.cpp vs vLLM vs MLX), quant engine, batch size, and system load. The relative rankings are more reliable than the absolute numbers.

Try the interactive tool →   Pick any GPU + model, switch quant and context, see real-time estimates.

MSRP is manufacturer list price. Amazon prices are whatever you see after clicking. Some links may earn us a commission at no extra cost to you (Amazon Associates). Missing a GPU or model? Send it in. Got real tok/s numbers from your setup? Share them and we'll add them as measured anchors.