llama.cpp has merged a router change that measures device memory before it autoloads a model, then unloads idle models until the newcomer fits.[1][2] A home server handling chat, embeddings and coding can now limit loaded models by gigabytes instead of merely counting them.
There is an availability wrinkle. The code landed in commit `85cc432` on October 27, but the public release list still stopped at b11927 when HighVRAM checked at 7:08 a.m. Central Time.[2][4] You can build the feature from source now. People who use llama.cpp’s packaged binaries should wait for a release whose commit includes `85cc432` rather than assuming b11927 has it.
Why model count was the wrong limit
llama.cpp’s router mode can expose several local models through one server and load the requested model automatically.[3] Before this change, `–models-max` limited how many models could stay loaded and evicted the least recently used model when that count was reached.[3] Four tiny models and four enormous models were treated alike, which is tidy bookkeeping and poor memory management.
The contributor’s example came from a roughly 120GB unified-memory machine.[1] An embedding model used about 6GB and a 35B Q6_K model used about 37GB, so the pair occupied roughly 43GB.[1] Adding an 81GB 122B Q4_K_M model brought the request to 124GB and could push the server out of memory.[1] HighVRAM recalculated those figures: 6 + 37 = 43GB, while 43 + 81 = 124GB, four gigabytes beyond a 120GB pool.
Setting the old limit to one avoided the crash, but it also forced the small embedding and chat models to take turns loading.[1] That is exactly the wrong trade for a local agent, which may need retrieval and generation during the same job.
The new flag is `–models-memory-margin N`, where `N` is the amount of device memory, in MiB, that the router should try to leave free on each device.[2] A value of zero disables memory accounting, and zero remains the default.[2]
For example:
“`bash llama-server –models-dir /path/to/models –models-memory-margin 2048 “`
That asks the router to preserve 2GiB of device headroom. It is an example, not a universal safe setting. Long contexts, other GPU applications and driver allocations can justify a larger margin.
What llama.cpp actually measures
When the flag is enabled, the router starts a short-lived child process for each model, reads its metadata without allocating the model and reports two memory estimates.[3] The minimum uses the smallest context that `–fit` may choose; the maximum reflects the largest context that fit when the measurement ran.[3] A loaded instance then reports its actual allocation as the effective figure.[3]
Before loading another model, the scheduler checks every GPU or other accelerator involved.[3] If the new model’s minimum estimate will not fit while preserving the requested margin, it evicts idle models in least-recently-used order.[3] A model handling a request is not pulled out from under it.[3]
This is unloading and reloading, not magic memory compression. The next request for an evicted model has to pay the load delay again, and the model files still need to live on storage. Fast NVMe will make that less annoying than a hard drive, but it cannot make an 80GB reload disappear.
HighVRAM checked out the exact merged commit and built `llama-server` on an Apple-silicon Mac. The resulting binary exposed `–models-memory-margin` with the documented per-device and LRU behavior. We also ran the two new upstream unit tests that cover model measurement and eviction; both passed. Those tests verify the scheduler paths with tiny fixtures. They are not proof that every large multi-GPU setup will avoid an out-of-memory failure.
What the memory check still misses
The guard only manages accelerator memory, while host CPU memory is treated as unlimited and a build without a GPU device gets no benefit from the feature.[3] A machine that offloads part of a model into system RAM can therefore exhaust host memory even while the device-side calculation looks acceptable.
Multimodal users need more caution. The initial estimator does not include the `mmproj` projector used for image or audio input, so a vision model needs more memory than the reported minimum.[3] Failed measurements fall back to loading without a memory check; if no other model is loaded and the newcomer’s minimum estimate still does not fit within available device memory after the margin, the router loads it anyway and lets the instance’s existing `–fit` behavior make the final attempt.[3] If every possible eviction candidate is busy, the autoload request waits rather than yanking a model out from under an active request.[2]
This update does not make a 40GB model fit wholly in 24GB of VRAM. llama.cpp can already run a larger model through partial CPU offload if the machine has enough system RAM; the new flag instead manages accelerator-memory contention among router-loaded models.[3] That makes one 24GB GPU, a 48GB workstation card or a large unified-memory Mac easier to share among models that already fit the intended placement individually. The clearest beneficiaries are local agents and household servers that keep a small embedding model warm but occasionally call a much larger coding or reasoning model.
If you run one model at a time, leave the new flag alone. If your router serves several models, update after `85cc432` reaches a packaged release, set explicit context sizes for the models that must coexist, choose a nonzero margin, and inspect the new `memory` fields from `GET /models` before trusting it unattended.[3] Real 24GB, 48GB and unified-memory systems still need workload testing to show that they stay responsive when a busy agent asks for the model that almost fills the machine.