Google's Gemma 4 generation covers the consumer sweet spot: five instruction-tuned models from 5.1B to 30.7B, including the 26B-A4B MoE variant. All are Apache-2.0 with 128K–256K context — sized for single-GPU local use.
Google's Gemma line is the open-weight sibling of its frontier models, and the current Gemma 4 generation is aimed squarely at the consumer sweet spot: five instruction-tuned models from 5.1B to 30.7B parameters, every one under the permissive Apache-2.0 license. That license matters — you can fine-tune, redistribute, and ship Gemma commercially without the usage restrictions that come with some rival families.
Two things define the family for local use. First, context is generous across the board: 128K on the two small models and 256K on the rest, so long-document RAG and agent traces are realistic on a single GPU. Second, the lineup includes a MoE variant, the 26B-A4B, which is currently the most downloaded Gemma 4 model on Hugging Face — it delivers big-model quality while only activating 3.8B parameters per token, so it runs much faster than a dense model of the same total size.
All VRAM figures below use our standard formula: Q4 weights + KV cache at 8K context + 1 GB overhead. Longer contexts add roughly 0.025 GB per active billion parameters per 1K tokens — the dense 31B grows fast, the MoE barely grows at all.
Size ladder
- Gemma 4 E2B (5.1B dense, 128K ctx) — Q4 weights just 2.81 GB, about 4.8 GB total at 8K context → comfortable on any 12 GB card, and even 8 GB gaming GPUs work at Q4. FP16 weights (~10.2 GB) also fit a 12 GB card. A snappy daily driver for chat, summarization, and embeddings-adjacent tasks. Model page
- Gemma 4 E4B (8B dense, 128K ctx) — Q4 about 7.0 GB all-in → the classic 12 GB GPU (RTX 3060 12GB, 4070 class) is the natural home. FP16 pushes to ~16 GB plus KV, so stick to Q4/Q8 on 12–16 GB cards. A clear quality step up from E2B for a modest VRAM cost. Model page
- Gemma 4 12B (11.95B dense, 256K ctx) — Q4 about 10.0 GB → fits a 12 GB card with little headroom for long context; a 16 GB card is the comfortable pick, leaving room for the 256K window to actually be used. Model page
- Gemma 4 26B-A4B (25.2B MoE, only 3.8B active per token, 128 experts / 8 routed, 256K ctx) — Q4 about 15.6 GB → a 16 GB card is borderline at 8K context, so 24 GB (RTX 3090/4090) is the realistic tier. The payoff: because only 3.8B is active, generation speed feels like a ~4B model while quality sits near the 31B. Its KV cache is also tiny — even at 256K context it adds under 25 GB. Model page
- Gemma 4 31B (30.7B dense, 256K ctx) — Q4 about 24.0 GB → a 24 GB card is exactly at the line at 8K context; for any real context length you want 48 GB (RTX A6000-class or dual-GPU). FP16 (~61 GB weights) belongs on 96 GB+ setups. The family's quality ceiling. Model page
Family highlights
- Apache-2.0 everywhere — no click-through licenses, no commercial-use carve-outs, and GGUF quants are easy to redistribute.
- Long context as standard — 128K minimum, 256K on 12B and up. Note that KV cache scales with active parameters, so the MoE model's long context is far cheaper in VRAM than the dense 31B's.
- A true MoE option — the 26B-A4B routes 8 of 128 experts per token. For local inference this is the family's best speed-per-quality point, and its modest active size means high tok/s even on consumer GPUs.
- Single-GPU by design — every model in the family fits one consumer or workstation card at Q4; nothing here requires a multi-GPU rig unless you insist on FP16 at the top end.
Which one should you pick
- 12 GB VRAM — E2B or E4B at Q4. E2B leaves the most headroom for context and a desktop environment; E4B is the better model if quality matters more.
- 16 GB VRAM — the 12B at Q4 is the sweet spot, with room for meaningful context. The 26B-A4B technically squeezes in at 8K but has no margin — treat 16 GB as MoE-with-Q2 territory, not Q4.
- 24 GB VRAM — the 26B-A4B at Q4 is the standout choice: near-31B quality, ~4B-class speed, and cheap long context. The dense 31B only fits here at 8K context with nothing to spare.
- 48 GB VRAM — the 31B at Q4 with real room for its 256K window, or the 26B-A4B at Q8/FP8 for maximum quality at high speed.
- 96 GB+ VRAM — FP16 31B, or running several Gemma instances side by side for agent workloads.
If you're unsure, start with the 26B-A4B on a 24 GB card — it's the family's community favorite for a reason.
Links
5 models in the Gemma family, grouped by series. Q4 (GB) is weights-only; total VRAM adds KV cache and overhead — the fit check links compute it for a representative retail GPU at 8K context.