How much VRAM do you need for a 70B LLM?

Short answer: a “70B model” does not have one VRAM requirement. The exact model file, quantization, context, cache precision, runtime buffers and concurrency determine the memory budget. As a practical starting point, 24GB and 32GB cards generally cannot hold a typical 4-bit 70B/72B model entirely in dedicated VRAM; 48GB can be possible but tight for some artifacts and contexts; larger dedicated or unified-memory tiers provide more room. Verify the exact artifact before buying.

70B VRAM requirements by precision

The raw weight calculation is parameters × bits per weight ÷ 8. It is useful for orientation, but real quantized files contain metadata and may mix precisions. It also excludes the KV cache and runtime allocations.

Nominal precision	Raw weights for exactly 70B parameters	What the number omits
2-bit	17.5GB	Format overhead, mixed precision, cache, buffers and quality effects
3-bit	26.25GB	Format overhead, mixed precision, cache, buffers and quality effects
4-bit	35GB	Format overhead, mixed precision, cache and runtime buffers
5-bit	43.75GB	Format overhead, mixed precision, cache and runtime buffers
8-bit	70GB	Format overhead, cache and runtime buffers
16-bit	140GB	Cache and runtime buffers

A real example: Qwen2.5-72B-Instruct GGUF

Qwen’s official Qwen2.5-72B-Instruct-GGUF repository identifies 72.7B total parameters, 80 layers, 64 query heads and 8 KV heads. Its repository currently lists these approximate download sizes:

Quantization	Published repository size	Single-GPU capacity implication
Q2_K	27.3GB	The file is below 32GB, but loading, cache and buffers still need space; fit is not established by download size.
Q3_K_M	35.5GB	Already larger than 32GB before cache and runtime allocations.
Q4_0	41.4GB	Below 48GB nominal capacity, but the remaining margin is limited.
Q4_K_M	44GB	A 48GB card is a tight target once cache and runtime allocations are included.
Q5_K_M	51.7GB	Larger than a 48GB pool before cache and runtime allocations.
Q8_0	77.5GB	Requires a larger memory path or offload/splitting.

Context can consume the remaining margin

Using Qwen’s published geometry, an illustrative FP16 KV-cache tensor payload for one sequence is:

2 × layers × KV heads × head dimension × token slots × bytes
2 × 80 × 8 × 128 × token slots × 2

At 8,192 token slots, that equals 2,684,354,560 bytes, or 2.5GiB. At 32,768 slots it becomes 10GiB. The leading 2 counts keys and values. This is tensor-payload arithmetic, not a runtime allocation measurement; padding, metadata, batching and implementation details can change the result. Cache precision can also differ.

This is why “the 44GB file fits in 48GB” is not a safe purchase conclusion. The 48GB label is decimal capacity (about 44.7GiB), while the file display is also decimal. After converting units and adding even the illustrative 8K cache, little room remains for runtime allocations. Measure the exact runtime configuration.

What each hardware tier means

• 24GB: expect CPU offload or a materially smaller/stronger quantization for most 70B-class artifacts. A 24GB card is not a full-GPU 4-bit 70B default.
• 32GB: some very small quantizations may load depending on runtime and context, but common 3-bit and 4-bit artifacts can already exceed the pool or leave too little working room.
• 48GB: plausible for selected 4-bit artifacts, but exact file size and context matter. Treat it as a configuration to validate, not a blanket guarantee.
• 64GB dedicated or unified memory: more room for many 4- and 5-bit paths, but unified memory is shared with the operating system and applications.
• 80–96GB and above: opens larger quantizations and context budgets, subject to software support and the distinction between dedicated and shared memory.

Can two 24GB GPUs run a 70B model?

Potentially, with software that explicitly partitions the model. Two 24GB cards remain two separate memory pools; installing them does not create one automatic 48GB GPU. llama.cpp documents layer, row and experimental tensor split modes plus per-device tensor-split controls. PCIe lanes, motherboard spacing, power, cooling and the chosen split all matter.

What about CPU offload?

llama.cpp and compatible applications can keep some layers in system RAM and some in VRAM. This can make a model usable on a smaller GPU, but it is not equivalent to holding the full workload in fast GPU memory. Measure prompt processing and generation on your intended split rather than assuming a fit result is a speed result.

A purchase checklist for 70B local inference

1. Choose the exact model repository and artifact—not only the “70B” label.
2. Record quantization, file size, context length, cache precision and concurrent sequences.
3. Check whether your runtime supports the GPU architecture or Apple-silicon backend.
4. Use the runtime’s pre-load estimate, then test a representative long request with normal background applications open.
5. Compare answer quality as well as fit and speed; a smaller quantization can change the result you care about.

Compare hardware paths

Use the hardware guide to compare 24GB, 32GB, 48GB and unified-memory systems. For consumer GPUs, see RTX 3090 vs 4090 vs 5090. If capacity above 32GB is the deciding factor, compare Mac Studio vs RTX 5090. Then check screened offers without treating a listing title as proof of workload fit.

Ready to turn the memory budget into a shortlist? Compare the best GPU and system paths for a 70B LLM across 48GB, 64GB, 80GB, 94GB, 96GB, 128GB and 141GB tiers without treating capacity as a speed ranking.

\n

Reviewed October 24, 2026. Calculations are illustrative and sources are linked. HighVRAM did not benchmark the cited model on every memory tier. Some marketplace links on related pages are affiliate links.

Deciding between common tiers? See 24GB vs 48GB VRAM for local AI for single-GPU, dual-GPU, ECC and buying tradeoffs.

Evaluating a one-GPU 96GB path? Read the RTX PRO 6000 Blackwell local-AI guide for 70B fit, bandwidth, power and edition caveats.

Considering a 64GB accelerator? Read the AMD Instinct MI210 local-AI guide before treating HBM capacity as a desktop-ready purchase.

Trying to fit a 70B model across two cards? Read the multi-GPU VRAM guide before adding capacity labels; the runtime decides how weights, KV cache and buffers are placed.

Choosing between 32GB and 48GB CUDA cards? The RTX 5090 vs RTX A6000 comparison explains why the 5090 cannot hold a 44GB artifact entirely in VRAM and why 48GB still requires overhead headroom.

Choosing the 70B artifact? Compare Q4_K_M, Q5_K_M, Q6_K and Q8_0 before treating bit labels as universal quality, speed or memory guarantees.

Using CPU offload for a 70B model? The CPU-offload guide explains why successful loading does not establish acceptable response time, context capacity or system-RAM headroom.

Sizing a Mac for 70B? The Mac Studio unified-memory guide explains why 48GB is tight, how 64GB changes the margin, and what 96GB/128GB add without promising speed.

Looking at a used A100 for a 70B-class model? Read the A100 40GB vs 80GB local-AI guide. The 40GB version can be too tight once the exact artifact, KV cache and runtime allocations are counted; the 80GB version still requires PCIe-versus-SXM and server-integration checks.

Evaluating an 80GB or 94GB Hopper accelerator? Use the NVIDIA H100 local-AI guide to separate PCIe, SXM and H100 NVL deployment requirements after calculating the complete 70B workload.

Planning beyond 96GB on one CUDA device? Evaluate the 141GB NVIDIA H200 local-AI tier only after calculating the complete model, KV-cache, runtime and headroom requirement.

Does a workload justify 141GB or 180GB per GPU? Use the H200 vs B200 local-AI comparison after calculating the exact artifact, KV cache, runtime allocations and required headroom.

Choosing between the tight 48GB tier and more headroom? Read 48GB vs 80GB VRAM for local AI after calculating the exact 70B artifact and KV cache.

\n

Comparing one 96GB workstation GPU with 80GB or 94GB Hopper? Read RTX PRO 6000 Blackwell vs H100 after calculating artifact, cache and runtime headroom.

Does a smaller model meet the workload? Use the GPU-memory-to-model-size map before committing to a 70B-class hardware tier.

Multimodal 72B case: the Qwen2.5-VL 72B VRAM guide adds a required GGUF projector, image/video workload memory and the official 146.822GB BF16 path.