LLAMA.CPP GPU GUIDE
llama.cpp GPU guide: VRAM, context, and tok/s
llama.cpp makes local inference portable, but a useful command still depends on the model fitting in memory and the GPU moving its weights fast enough.
Start with a model that fits
The weight file is not the whole budget. KV cache grows with context and the backend needs runtime overhead, so use a fit calculation instead of comparing file sizes alone.
Quantization trades memory for precision
Q4, Q8, and FP16 change weight memory and usually change the set of models your GPU can run. The calculator keeps the quantization choice live while you explore.
Bandwidth is the speed constraint
Decode speed is estimated from memory bandwidth and model size, with measured anchors replacing the estimate when the catalog has an exact match.
Ready to test your machine?
Enter a GPU or usable VRAM and the calculator will rank models for your workload, quantization, and context.
Open the calculator