A new llama.cpp pre-release makes long local-AI conversations much faster on Intel’s Arc B580, at least in the test submitted with the code.[2][5] With Llama 3.1 8B and a 16K-token context, generation rose from 31.3 to 54.2 tokens per second after the runtime began processing a quantized KV cache directly on the GPU’s XMX matrix engines.[2]
That is a 73.2% gain by HighVRAM’s calculation from the submitted rates.[2] A 128-token reply would take about 2.36 seconds at the new rate instead of 4.09 seconds, before allowing for prompt processing and other overhead.[2]
The software is real and downloadable in packaged builds.[5] The performance result still comes from one project contributor, not an independent review.[2]
For Arc owners, the distinction matters. This isn’t another peak-TOPS promise. It changes the slow part of an actual long-running chat.
Why the speedup grows with the conversation
A local model keeps information about earlier tokens in a key-value cache, usually called the KV cache. As the conversation grows, the model has more cached data to read during each new generation step.[2] Quantizing that cache reduces its memory footprint, but the GPU still has to turn the compressed values into useful math.[2]
Before this change, llama.cpp’s Intel SYCL paths either dequantized cache values with scalar operations or converted the cache to 16-bit values for a call.[2] The new kernel performs the attention calculation directly against Q4_0 or Q8_0 cache blocks on Intel’s XMX hardware, while keeping a later part of the operation in 16-bit form.[2]
It covers token generation and small batches of up to 31 query tokens.[2] Prompt ingestion stays on the existing path.[2]
The submitted benchmark shows exactly where that helps. At an empty context, the gain was only 3.6%.[2] At 16K tokens, generation jumped 73%, while prompt processing was flat.[2] This is a long-context optimization, not a magic 73% sticker for every workload.
The test used a 4-bit Llama 3.1 8B model with both the key and value caches set to Q4_0.[2] Hardware and software were an Arc B580, Core i5-13400, Ubuntu 24.04, Intel’s icpx 2025.3.3 compiler and NEO 26.35.39758.10 driver.[2] HighVRAM has not reproduced the test on Intel hardware, and the pull request does not provide an output-quality comparison between the quantized and 16-bit caches.
The memory trade is useful too
The code definitions make the storage difference concrete. A Q4_0 block stores 32 values in 18 bytes: 16 bytes of four-bit values plus a two-byte scale.[6] Q8_0 uses 34 bytes for the same 32 values.[6] That works out to 4.5 bits per value for Q4_0 and 8.5 bits for Q8_0.[6]
Compared with a raw 16-bit cache, those formats reduce the cache data by about 71.9% and 46.9%, respectively.[6] That is HighVRAM’s calculation from the published block structures, before allocator padding or runtime overhead.[6] It doesn’t shrink the model weights. It does leave more of a small GPU’s memory available for context, model layers and concurrent work.
Intel’s 12GB card is where that memory saving gets interesting. Intel specifies 12GB of GDDR6 and 456GB/s of memory bandwidth, with a recommended customer price of $249.[4] The list price is not a current retail quote, but 12GB gives local model weights and their working memory useful breathing room.
Twelve gigabytes still imposes a hard limit. Model weights, the KV cache and runtime buffers all have to share it. The new kernel makes a supported workload faster and can reduce cache storage. It does not turn 12GB into 24GB, however enthusiastically the release notes are read.
Available now, with pre-release strings attached
The change is merged and packaged in llama.cpp build b11980.[2][5] GitHub marks that build as a pre-release, and its assets include Ubuntu SYCL packages in FP16 and FP32 plus a Windows SYCL archive.[5]
Setup differs by operating system. llama.cpp’s documentation says the Windows package bundles the SYCL runtime.[3] The Ubuntu packages require compatible Intel GPU drivers and matching oneAPI components already installed.[3] People who want an appliance-like install should wait for their preferred front end to absorb the change. People already maintaining a llama.cpp SYCL setup can test it now.
The fast path targets Intel’s Xe2 generation: Arc B580 and B570, Arc Pro B70, Lunar Lake and Panther Lake.[2] The broader SYCL backend supports older Arc hardware, but the project’s list for this kernel does not include Arc A770 or A750.[2][3] Owners of those cards should not assume the benchmark transfers.
Testers can disable the new path with `GGML_SYCL_FA_DPAS=0`, which permits an A/B comparison on the same machine.[2] A useful test should keep the model, prompt, context, cache types, driver and build fixed, then compare both speed and output behavior.
Current B580 owners running long chats or retrieval-heavy prompts should try the pre-release with Q4_0 and Q8_0 cache settings. A buyer should wait. One contributor benchmark is enough to make a free update interesting, but not enough to make a graphics-card purchase. Results across Windows, more models and ordinary home systems will decide whether 73% is a repeatable gain or a particularly flattering test case.