tandem.cpp is a new llama.cpp patch set for running mixture-of-experts models that spill beyond GPU memory. Its developer reports 97.3 tokens per second from Qwen3.8-Flash-Next on an RTX 4090 with 64GB of system RAM, up from 29.8 tokens per second in the llama.cpp baseline.[1] That is a 3.27-times speedup by HighVRAM’s calculation, and the model stack is much larger than the card.
The code is available now under the MIT license, and HighVRAM verified that the 432,439-byte patch applies cleanly to the exact llama.cpp commit named by the project.[1] The speed results all come from tandem.cpp’s developer, however. Nobody independent has reproduced them yet.
For home-AI builders with a 24GB or 48GB Nvidia card and plenty of ordinary RAM, this is exactly the awkward hardware gap that needs work. Large sparse models can be fast once loaded because each token activates only part of the network. Fitting their complete collection of experts into VRAM is another matter.
What tandem.cpp changes
Standard llama.cpp can split model layers between GPU memory and system RAM. tandem.cpp takes a more active approach to the expert layers inside mixture-of-experts models: it keeps all experts in host memory, fills unused VRAM with a cache of recently used experts, and handles a cache miss by dividing work between the CPU and a PCIe upload to the GPU.[1] The split is based on measured CPU and PCIe speeds, according to the project.[1]
That matters because CPU offload is usually discussed as a capacity trick. It makes a model fit, but often leaves the GPU waiting while system memory dribbles weights across the bus. tandem.cpp tries to use the CPU and GPU at the same time instead of making one wait for the other.
The project remains a patch, not an upstream llama.cpp feature. Its build script clones llama.cpp at commit `f5222f25c`, applies the patch and compiles `llama-server`.[1] The documented path requires Linux, CUDA and a source build. The developer tested Ubuntu 22.04 and 24.04 with CUDA 13.0 through 13.4 on RTX 4090, RTX 6000 Ada and RTX Pro 6000 Blackwell cards.[1] Windows, macOS, AMD GPUs and Intel GPUs are not supported test targets.
The 97-token result needs about 80GiB of model files
The headline test used an Intel Core i5-12400F, 64GB of DDR4 measured at 43GB/s, an RTX 4090 on PCIe 4.0 x16, and one user session.[1] On the first 200 GSM8K questions, Qwen3.8-Flash-Next IQ3_S with a multi-token-prediction helper rose from 29.8 to 97.3 tokens per second with greedy decoding. With the model’s recommended sampling settings, it rose from 29.4 to 96.1 tokens per second.[1]
The downloads put those numbers in perspective. The two IQ3_S model shards total 83,617,662,656 bytes, or 77.88GiB after dividing by 1,073,741,824.[3] The speculative-decoding helper adds another 2,664,610,336-byte GGUF, or 2.48GiB.[4] Together they occupy about 80.36GiB before runtime overhead, KV cache or filesystem slack. A 24GB GPU plainly cannot hold that stack, and even 64GB of system RAM cannot keep every byte resident at once.
The project says both tandem.cpp and the baseline used memory-mapped files for this consumer-PC test because the models exceeded available RAM.[1] That makes the claimed result unusually interesting. It also makes storage behavior part of the benchmark, so a different SSD, page-cache state or operating-system load could move the number.
The improvement was not confined to one model. In the same machine, gpt-oss-120b rose from 27.8 to 51.5 tokens per second with greedy decoding and from 27.7 to 48.0 with recommended sampling.[1] Those are 1.85-times and 1.73-times increases by HighVRAM’s calculation. The range is a useful warning against treating 3.27 times as a general multiplier.
Where it helps, and where it does not
The best buyer is an experienced Linux user who already owns an Nvidia GPU, has at least 64GB of RAM, and wants to run a sparse model that is too large for VRAM. This software does not turn a 24GB card into 80GB of VRAM. It makes the boundary between GPU, CPU, RAM and storage less wasteful.
A model that already fits gains little. On the project’s 48GB RTX 6000 Ada test, a 22.4GB Qwen3.6 model improved from 173.1 to 179.2 tokens per second, roughly 4%.[1] tandem.cpp also targets one user at a time; the developer says concurrent requests work but scale less efficiently and can reduce total throughput when the expert cache is small.[1]
Running far beyond system RAM can erase the benefit. DeepSeek-V4.1 was about three times larger than RAM on the 64GB consumer machine, and tandem.cpp produced 5.99 tokens per second against llama.cpp’s 6.86.[1] That is a 12.7% slowdown. The project’s own explanation is that its CPU path does not yet take cache misses that must come from SSD storage.[1]
Quality also needs an asterisk. The project says its default path computes every routed expert and changes results only through floating-point rounding and kernel choices, while optional expert-skipping modes are off by default.[1] Its internal accuracy tests found results within run-to-run noise, but those tests were designed and run by the same person who wrote the patch.[1]
Local execution keeps prompts and model traffic on the owner’s machine, which preserves the privacy and control advantage of llama.cpp. The cost is operational rather than subscription-based: a source build, Linux, an Nvidia driver stack, enough RAM, and careful context sizing. The README estimates that Qwen3.8 consumes about 0.7GiB of additional KV cache for each extra 16,000 tokens of context, memory that comes out of the GPU expert cache.[1]
If that description sounds like a fun Saturday, tandem.cpp is available to try. Start with the exact pinned commit and benchmark one model against plain llama.cpp on your own prompts. Everyone else should wait for an upstream implementation, packaged binaries or an independent reproduction on common 24GB systems.
A repeatable comparison across 32GB, 64GB and 128GB hosts should come next, with cold and warm storage caches reported separately. That will show whether tandem.cpp can become a better default for oversized local MoE models or remains a clever patch for carefully tuned machines.