Llama-3.1-70B-Instruct

Meta · 70.6B · ctx 128K · Llama 3.1 Community · HuggingFace

Llama-3.1-70B-Instruct (Meta) is a 70.6B parameters open-weight model tracked on AI Hashrate. At 8K context we estimate roughly 53.83 GB VRAM for Q4 (weights 38.83 GB + KV 14.0 GB + 1 GB overhead) and 156.2 GB for FP16. About 11 GPUs/accelerators in our catalog fully fit this model at Q4 under that context assumption. Fastest Q4 configs: B200 192GB SXM ≈ 520.0 tok/s (measured); MI300X 192GB ≈ 48.2 tok/s (estimated); H200 141GB SXM5 ≈ 43.6 tok/s (estimated); H100 80GB SXM5 ≈ 30.5 tok/s (estimated); Gaudi 3 128GB ≈ 29.8 tok/s (estimated). Use the table below for VRAM fit and tok/s per dollar (list MSRP). Estimates follow memory-bandwidth math; measured rows override when present. See methodology for details. Methodology.

49 hardware configs — sorted by tok/s. Default fit context 8K.