Running Large Models on Consumer GPUs: Offload, Speed Decay and MoE

Why "it doesn't fit" doesn't mean "it won't run" — and what it costs you · AI Hashrate guide

Every local-LLM user eventually tries to run a model that's bigger than their VRAM. The good news: modern runtimes (llama.cpp, Ollama, LM Studio and friends) will happily offload — splitting the model between GPU and CPU memory — so it still runs. The bad news: speed falls off much faster than intuition suggests. This guide explains the mechanics, the (VRAM ÷ need)² decay we use in our estimates, and why MoE models are the cheat code for big models on small hardware.

What is GPU offload, exactly?

Decode is a memory-bandwidth game: to generate one token, the hardware must read every active weight once. When the whole model sits in VRAM, that read happens at GPU memory speed — 1,000+ GB/s on an RTX 4090. When it doesn't fit, the runtime keeps as many layers as possible in VRAM and leaves the rest in system RAM, shuttling them across PCIe (tens of GB/s) every single token.

So an offloaded model is a hybrid: part of each token's weights arrive at VRAM speed, the rest crawl over PCIe, and the slowest part sets the pace. It runs — but the bottleneck is no longer your GPU.

Why does speed decay with (VRAM ÷ need)²?

Our estimator uses a deliberately conservative rule: when a model doesn't fit, the utilization factor drops from 0.35 (fits) to (VRAM ÷ need)², floored at 0.01. Why squared? Because two penalties compound:

• The fraction penalty: only VRAM ÷ need of the model travels at GPU speed; the rest crosses PCIe. Speed scales roughly with the resident fraction.
• The efficiency penalty: offloaded runs lose the batching and caching efficiencies that make resident inference fast — layer swaps, synchronization stalls, and CPU-side attention work all eat throughput. Empirically, real offload numbers land well below the naive linear fraction, so we square it.

Worked example — Llama-3.1-70B Q4 on a 24 GB RTX 4090: the model needs 53.8 GB at 8K context, so the ratio is 24 ÷ 53.8 = 0.45, squared = 0.20, versus 0.35 if it fit. Estimated speed: ~5 tok/s instead of the ~12 tok/s a linear guess would suggest (est.). Push to a denser case — 70B FP16 on the same card needs 156 GB — and the estimate collapses to ~0.2 tok/s (est.): technically running, practically unusable.

The 32B row is the important lesson: just missing the fit line keeps most of your speed — 0.88 vs 0.35 would actually beat the fit factor in our model, because the formula caps the penalty at the resident fraction. Deep misses are catastrophic; shallow ones are survivable.

Why are MoE models the offload sweet spot?

Mixture-of-Experts models store many expert networks but only activate a few per token. That splits the problem in two:

• VRAM depends on total params — all experts must be stored. Qwen3-30B-A3B needs 18.4 GB at Q4, similar to a dense 14B.
• Speed depends on active params — only ~3B params are read per token. On an RTX 4090 that's an estimated ~214 tok/s at Q4 (est.) — a 30B-class model answering faster than most dense 8B setups.

And when an MoE model does offload, the damage is muted: the PCIe shuttle only moves the active experts, not the whole model. This is why giant MoE flagships are surprisingly usable on unified-memory machines: gpt-oss-120b (8B active) on a 128 GB M4 Max estimates ~43 tok/s at Q4, and Qwen3-235B (22B active) on a 192 GB Mac Studio M2 Ultra estimates ~23 tok/s (est.). A dense model of the same total size would be a slideshow on the same box.

What are your practical options when a model doesn't fit?

• Shorten the context. KV cache is linear in context; dropping 8K → 4K can reclaim several GB on mid-size models and flip a "miss" into a "fit".
• Stay at Q4. At 0.55 bytes/param it is the only tier that keeps 30B-class models within reach of consumer cards; see our quantization guide.
• Prefer MoE when capacity is tight. Small active params keep both KV cache and per-token reads low.
• Accept offload only for batch work. ~5 tok/s is fine for overnight summarization, painful for chat.
• Add capacity, not compute. A second-hand 24 GB card or a unified-memory Mac adds usable VRAM — the actual constraint. Check combinations on the AI Hashrate matrix before spending.

How should you read our "fits / doesn't fit" line?

We mark a combination as fitting only when weights + KV(8K) + 1 GB overhead is ≤ 95% of VRAM, and we apply full 0.35 utilization only there. "Doesn't fit" never means "won't run" — it means "runs offloaded, at the decayed estimate shown". The full formula and its limits are documented on the methodology page.