CPU offload can make a local LLM fit when it exceeds GPU VRAM, but it does not turn system RAM into GPU VRAM. A compatible runtime can keep selected model tensors on the CPU and place the rest on the GPU. That expands the usable memory path at the cost of CPU work, host-memory traffic and transfers between devices.
See the quick answer · Plan memory · Test whether it is usable
Fit is not speed. A model that starts with partial CPU placement may still miss your time-to-first-token or generation target. Treat offload as a configuration to measure, not proof that you no longer need more VRAM.
What does CPU offload mean for a local LLM?
In a hybrid CPU/GPU runtime, some model work and data stay in system RAM for CPU execution while supported portions are offloaded to the GPU. The exact unit may be layers, tensors, feed-forward or expert weights, and the runtime decides where cache and intermediate allocations live. “CPU offload” is therefore not one universal mode.
| Question | Answer | What decides it |
|---|---|---|
| Can CPU offload reduce required VRAM? | Often, for a supported runtime | Which tensors or layers remain on CPU and what must still be allocated on GPU. |
| Does the model still need system RAM? | Yes | Weight placement, memory mapping, runtime buffers, OS and other applications. |
| Will it run as fast as full GPU placement? | No guarantee | CPU speed, memory bandwidth, device transfers, kernels and workload. |
| Does successful loading prove the context fits? | No | KV cache, concurrency, buffers and later peak allocations. |
| Is every app able to offload partially? | No | The exact runtime, backend, build and frontend controls. |
How llama.cpp exposes placement
The official llama.cpp README identifies CPU+GPU hybrid inference as a way to partially accelerate models larger than total VRAM capacity. Its current server documentation exposes --n-gpu-layers to set the maximum number of model layers stored in VRAM, including automatic and all-layer modes.
The same server documents more specific controls: selected dense feed-forward layers can remain on CPU, and Mixture-of-Experts weights can be kept partly or fully on CPU. Those options have different communication and memory behavior. Do not copy a layer count from another model without checking the current build and its startup report.
Offload, memory mapping and swap are different
- GPU offload: supported model data and computation are assigned to the accelerator.
- CPU placement: data remains in host memory and applicable computation runs on the CPU.
- Memory mapping: the operating system maps the model file into virtual memory; resident pages can still consume physical RAM.
- Swap or compression: the OS moves or compresses memory under pressure. This is not a designed high-performance offload tier.
llama.cpp documents memory-mapped and locked loading modes separately from device placement. A model file being memory-mapped does not mean it needs no RAM, and a machine avoiding an immediate out-of-memory error does not establish stable performance under sustained use.
Plan VRAM and system RAM separately
Start with the exact GGUF or other artifact, not only its parameter count. Then account for:
- the portion of model weights assigned to each device;
- KV cache at the intended context length and concurrency;
- runtime buffers and compute workspace;
- the operating system, frontend and other applications in system RAM;
- page cache, memory mapping and a margin for peak allocations;
- additional models such as embeddings, vision projectors or speculative-draft models.
Do not subtract GPU VRAM from file size and call the remainder the required system RAM. Placement is not necessarily byte-for-byte, and some allocations may be duplicated or fixed to a particular device. Use the LLM VRAM calculator as a planning range, then observe the real runtime.
What happens to the KV cache?
KV-cache placement is runtime- and mode-specific. It can consume a material amount of memory as context or concurrent sequences grow, and it may not follow model weights in the way you expect. Read the KV-cache VRAM guide, verify the runtime’s cache controls, and inspect memory after the intended context is actually allocated.
How to check placement
For llama.cpp, read the startup log and record the model, backend, device list and number of offloaded layers. Do not infer placement from GPU utilization alone. For Ollama, the official FAQ documents ollama ps: its PROCESSOR field reports whether a model is loaded on GPU, CPU or both. “100% GPU” describes placement, not spare VRAM or utilization.
Also observe system RAM, GPU memory, memory pressure or swap, and temperatures before loading, after loading and during the demanding phase. A single idle snapshot can miss the peak.
Test whether CPU offload is usable
- Fix the model artifact, quantization, context, prompt, output length and runtime version.
- Record the automatic placement chosen by the runtime.
- Measure model-load time, time to first response and generation separately.
- Repeat with fresh sessions; label the first run instead of averaging it invisibly.
- If supported, change one placement control at a time and record GPU/CPU memory.
- Test the longest real prompt and the concurrency you intend to use.
- Stop if the system enters sustained memory pressure, unsafe temperatures or unstable behavior.
Use the local-LLM benchmark method to define pass/fail criteria. A configuration is useful only if the answer quality and wait meet your actual task—not because every layer technically loaded somewhere.
CPU offload versus more VRAM
| Path | Main advantage | Main cost |
|---|---|---|
| Partial CPU offload | Can use an existing GPU for a model beyond its VRAM | More host computation/traffic and runtime-specific tuning |
| Smaller quantization | Reduces weight memory while preserving full GPU placement when it fits | Potential task-quality tradeoff; format support varies |
| Two GPUs | Can distribute a supported workload across device memories | Separate pools, communication and platform complexity |
| One larger-VRAM GPU | Simpler single-device placement | Higher card and possibly system cost |
Compare GGUF quantizations, read why two GPUs do not automatically pool VRAM, and use the hardware guide when measured offload performance fails your requirement.
Reviewed October 24, 2026. Runtime options and defaults change. HighVRAM has not independently benchmarked CPU offload; this guide explains documented placement mechanisms and a test method, not a universal performance estimate.