llama.cpp has added FP8 key-value cache support, giving home users a new way to halve the memory consumed by long conversations and large document prompts.[1] The saving is real, but GPU support is not finished.
HighVRAM tested the current macOS arm64 binary, build b12283, with a small SmolLM2 model at an 8,192-token context.[3][5] On CPU, switching both caches from F16 to F8_E4M3 reduced the reported KV allocation from 180 MiB to 90 MiB. The same setting crashed during startup when the cache was placed on Apple’s Metal backend.
That makes this a useful release for CPU inference and an important foundation for CUDA, but not yet a setting Mac owners should turn on casually. A memory option that aborts the process is less “optimization” and more “surprise quit button.”
Why the cache can become the expensive part
A local model stores key and value tensors for tokens it has already processed. This KV cache stops the model from recomputing the whole conversation for every new token, but it grows with context length. Model weights stay roughly fixed; the cache keeps expanding as a chat, codebase or document collection gets longer.[4] That growth is the problem FP8 attacks.
The new llama.cpp path stores those tensors in OCP FP8 E4M3, an eight-bit floating-point format, instead of the usual 16-bit F16 or BF16 representation. In raw storage terms, one byte replaces two.[1][4] The merged implementation supports optional static per-tensor scales and includes end-to-end tests for conversion, loading, context shifts and scaled-cache correctness.[1]
vLLM already uses this idea. Its documentation describes FP8 KV cache as a way to reduce memory use, hold more tokens and improve serving throughput.[4] It also recommends calibrated scales for the best accuracy and warns that some attention layers are more sensitive than others.[4] That supports the memory argument, but it does not prove that every llama.cpp model will preserve quality at every context length.
The practical savings depend on model architecture. HighVRAM calculated the figures below from each model’s layer count, number of KV heads and head dimension.[6][7] The method is: two caches, multiplied by layers, KV heads, head width, context tokens and bytes per value.
For Llama 3.1 8B, a 32,768-token cache works out to 4 GiB in F16 or BF16 and 2 GiB in FP8.[6] At the model’s full 131,072-token context, the estimate falls from 16 GiB to 8 GiB.[6]
Llama 3.1 70B is more dramatic. Its eight KV heads keep the cache smaller than full multi-head attention would, but 80 layers still add up.[7] At 32,768 tokens, the estimate drops from 10 GiB to 5 GiB.[7] At 131,072 tokens, it falls from 40 GiB to 20 GiB.[7]
Those are cache-only calculations. Add model weights, compute buffers, the operating system and any memory reserved by the GPU runtime. FP8 does not make a 70B model fit on an 8GB card. It can, however, turn a long-context configuration that barely misses available memory into one that fits.
What HighVRAM verified
The b12283 release page provides a downloadable Apple-silicon binary.[3] Its command-line help lists f8_e4m3 as an allowed type for both the key and value caches. That establishes availability rather than treating a merged pull request as a finished release.
HighVRAM downloaded that binary and the public 96 MiB Q4_0 build of SmolLM2-135M-Instruct.[3][5] With an 8,192-token context and CPU-only execution, llama.cpp reported a 180 MiB F16 KV cache. Repeating the run with F8_E4M3 produced a 90 MiB cache and completed normally. This was a compatibility and allocation check, not a model-quality benchmark; a 135M-parameter model cannot tell us whether a 70B assistant loses accuracy after 100,000 tokens.
The Metal run failed before generation. llama.cpp reported that a preallocated cache tensor in a Metal buffer could not run SET_ROWS, then aborted. The binary still advertises FP8 as an allowed cache type, so Mac users do not get an early, friendly rejection.
Nvidia users are closer to the intended path. A separate CUDA implementation merged into llama.cpp on November 11 and adds OCP FP8 operations to that backend.[2] It landed after the commit used for b12283, however, so the newest downloadable binary HighVRAM inspected does not contain that CUDA work.[2][3] Building current source may expose it sooner, but HighVRAM has not tested performance or correctness on an Nvidia GPU.
Support is narrower than the option’s presence suggests. The FP8-cache pull request excluded model architectures that access KV data through paths the first implementation does not cover.[1] The project also had to account for scaled caches and backend-specific storage operations. That is normal for a feature touching the hottest part of inference, but it argues against flipping the switch across a model library and hoping for the best.
Who should use it now
CPU inference users who need long context and are constrained by system RAM have the clearest case. Start with a repeatable prompt and compare output against F16 or BF16 before adopting FP8 for important work. The memory reduction should be exactly 50% for the cache itself; anything less means another allocation is dominating the total.
Nvidia owners should wait for a release binary that includes the merged CUDA implementation unless they are comfortable building and validating main themselves.[2] AMD, Intel and Apple users should not assume that a generic command-line option means their accelerator backend supports every required FP8 operation.
Mac owners should leave the cache at its default for GPU-offloaded inference on b12283. CPU mode works in HighVRAM’s small compatibility test, but giving up Metal acceleration to save cache memory is a poor trade for most interactive use.
The next build matters more than the pull request. It needs the CUDA code, a clean unsupported-backend error instead of an abort, and quality tests at genuinely long contexts. Once those arrive, FP8 cache could be one of llama.cpp’s most useful memory controls. Today it is half a breakthrough, in the unusually literal sense.