KV Cache VRAM: Why 4-Bit Models Still Run Out of Memory

A local model can fit comfortably in memory and still exhaust it when you give it a long document.[1] The missing number is usually the KV cache: working memory that grows with every token in the prompt and response.[6]

Model-download pages make weights easy to compare.[7] A Q4 file is about half the size of an 8-bit file, so the buying advice often stops at “the model fits.”[7] That is necessary, but it is not enough. The runtime also needs memory for the context, compute buffers, and its own overhead.[1][6] A longer context can consume the memory margin that looked comfortable from the model file alone.[1]

You can estimate the biggest moving part before downloading anything.

The cache is not the model

Autoregressive models generate one token at a time.[6] Rather than recalculate the attention keys and values for every earlier token on every step, the runtime stores them in a KV cache and reuses them.[6] Hugging Face documents the cache as separate key and value tensors for each model layer, with dimensions for batch size, KV heads, sequence length, and head dimension.[6]

For a full-attention decoder with the same KV geometry in every layer and a batch size of one, a useful first estimate is:

KV bytes = 2 × layers × KV heads × head dimension × context tokens × bytes per value

The leading 2 is for keys and values.[6] FP16 and BF16 use two bytes per value.[2][6] The estimate is deliberately narrow: it does not include model weights, temporary compute buffers, runtime allocations, a desktop’s display use, or memory reserved by the operating system. Treat it as the floor for the cache, not a promise that the whole workload will fit.[1]

Context means more than the text you paste. System instructions, chat history, tool output, retrieved documents, and newly generated tokens all occupy the same sequence.[6] A 32K setting reserves or permits space for the conversation as a whole, depending on the runtime’s cache design.[1][3]

Two 7B-to-8B models, very different bills

Qwen2.5-7B-Instruct is a clean example. Its configuration lists 28 layers, four KV heads, 28 attention heads, a 3,584-wide hidden state, and a 32,768-token maximum position setting.[2] Qwen’s model card advertises 131,072 tokens through YaRN scaling, but says the checked `config.json` defaults to 32,768 and warns that static scaling can hurt shorter-text performance.[8] The calculation below therefore uses the default 32K configuration, not the extended mode. Dividing the hidden size by the attention-head count gives a head dimension of 128.[2]

Put those figures into the formula and its BF16/FP16 cache costs 56KiB per token.[2][6] That is about 448MiB at 8,192 tokens and 1.75GiB at 32,768 tokens.[2][6] These are HighVRAM calculations from the published configuration, not measured allocations.

A commonly used Q4_K_M GGUF for the same model is 4.68GB on disk, or about 4.36GiB.[7] Add the full 32K cache and the simple subtotal reaches roughly 6.11GiB before runtime overhead.[2][6][7] An 8GB GPU would have only about 1.89GiB left after that raw subtotal, before runtime and display overhead.[1][2] Whether it fits depends on the backend, offload split, cache type, and what else occupies the card. Treat 12GB as a safer planning target for this 32K example, not as a measured minimum; 16GB mainly buys breathing room.

Now compare Llama 3.1 8B. Meta lists a 128K context window and grouped-query attention for the family.[5] Nvidia’s published configuration for its Llama 3.1 8B FP8 conversion gives 32 layers, eight KV heads, and a 128-wide head dimension.[4] At two bytes per cache value, that works out to 128KiB per token: 1GiB at 8K, 4GiB at 32K, and 16GiB at 128K.[4][6]

Despite similar parameter counts, Llama’s maximum-context cache bill is much larger in this comparison.[2][4][5]

This is why “supports 128K” should never be read as “128K fits on the hardware that loads the weights.”[4][5] The context-window label describes a model capability. Your usable context is the smaller limit imposed by the model, runtime, available memory, and acceptable speed.[1][3]

Quantizing the cache helps, with a catch

Weight quantization and cache quantization are separate decisions.[1][3] Loading Q4 weights does not automatically make the KV cache four-bit.[3][7] Hugging Face’s Transformers documentation offers a quantized cache specifically to cut memory use, while warning that cache quantization can hurt latency when the context is short and memory is already sufficient.[1]

llama.cpp exposes the same choice more directly.[3] Its current command-line arguments include `–ctx-size`, `–cache-type-k`, and `–cache-type-v` (`-c`, `-ctk`, and `-ctv`).[3] A practical starting command for the Qwen example is:

llama-server -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M \
  -c 8192 -ctk q8_0 -ctv q8_0

That asks for an 8K context and 8-bit key and value caches.[3] In ideal storage terms, an 8-bit cache is about half the size of a 16-bit cache, while a 4-bit cache is about one quarter. Actual allocations can differ because runtimes add metadata, alignment, and temporary buffers. Backend support varies, and cache quantization can trade latency for capacity, so test both speed and answers before making it the default.[1][3]

There is another escape hatch: cache offloading.[1] Transformers can move most layer caches to system RAM and prefetch the active layer, but its documentation warns of a possible throughput penalty from moving those states back and forth.[1] On a desktop with plentiful RAM and a smaller GPU, that can be a reasonable way to read a long document privately without buying a new card. It will not make PCIe behave like VRAM.

What to set on a home machine

Start with the context you actually use, not the maximum printed on the model card. For ordinary chat and coding help, 8K is a sensible baseline. Move to 16K or 32K only when your prompts, repository extracts, or document jobs need it.

Watch the runtime’s startup log or memory meter after loading the model, then test with a prompt near the intended length.[1][3] Do not judge fit from an empty chat window.[1] Weight-memory savings are fixed when the model loads; KV-cache savings grow with context length.[1][6][7] If memory is tight, reduce context first, then test an 8-bit cache, 4-bit cache, or CPU offload as your backend allows.[1][3] A smaller weight quant may save more memory overall, but it also changes the model used for short and long sessions alike, so compare output quality before keeping it.[7]

The buying rule is simple enough to keep: budget for weights, the KV cache at your real context length, and a safety margin.[1][6][7] If a seller or model page gives you only the first number, the arithmetic is not finished.

Sources

Applying this to a 70B model? Read the 70B LLM VRAM requirements guide for an exact 72B artifact example and 8K-versus-32K cache arithmetic.