How to Run Qwen3.8-27B Locally with llama.cpp or Ollama

Short answer: the simplest consumer-hardware path is the separately maintained Unsloth GGUF conversion with a current llama.cpp build or Ollama. Start text-only with UD-Q4_K_M, a 16.464GB model file, and a modest context before enabling vision, long context, or MTP. Qwen’s official standard and FP8 checkpoints use different formats and substantially more memory; use the official Transformers, SGLang, vLLM, or TokenSpeed paths when that is the artifact you intend to deploy.

Choose the artifact before choosing the runtime

Path	Artifact	Useful starting hardware	What this guide establishes
llama.cpp or Ollama	Unsloth community GGUF, UD-Q4_K_M: 16.464GB	24GB practical consumer starting tier; 32GB gives more context and runtime headroom	Current repository commands and a bounded smoke-test workflow
Official FP8	Qwen FP8 safetensors: 30.867GB	48GB experimental; 64GB safer	Capacity boundary only; use a runtime supporting the official checkpoint
Official standard	Qwen standard safetensors: 55.563GB	80GB clearer single-device tier	Capacity boundary only; not a consumer GGUF workflow

Read the Qwen3.8-27B VRAM requirements guide for the complete 16GB-to-80GB table. Do not treat “the file downloads” as proof that the complete workload fits.

Apple-silicon path: use the Qwen3.8-27B Mac unified-memory guide to account for macOS, shared graphics memory, Metal state, cache and swap before choosing 24GB, 32GB, 48GB or 64GB.

Path A: run the community GGUF with llama.cpp

The official Qwen project points local users toward GGUF models for llama.cpp, and the Unsloth repository publishes direct -hf

Install and start the server

curl -LsSf https://llama.app/install.sh | sh
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

On Windows, the repository documents:

winget install llama.cpp
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

If you already use a downloaded llama.cpp release:

./llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

The command can download a remote artifact, so confirm the resolved filename, revision, byte size, and available disk space before treating the run as reproducible. Pin or record the revision for any benchmark.

Perform a text-only smoke test first

1. Close memory-heavy applications and record available accelerator and system memory.
2. Start with one request and a modest context allocation rather than the advertised maximum.
3. Send a short deterministic prompt whose answer you can inspect.
4. Record load success, fallback/offload, peak accelerator memory, peak system memory, errors, and response correctness.
5. Only then increase context or add additional requests.

Qwen documents a native 262,144-token context and extension up to one million tokens for the model family. That is a model capability, not evidence that your 24GB or 32GB configuration can allocate the maximum. Cache, runtime state, vision inputs, MTP, and concurrency all consume additional memory.

Path B: run the same community GGUF with Ollama

The current Unsloth repository exposes this direct Ollama command:

ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

This uses the community GGUF, not Qwen’s official FP8 checkpoint. After the download, record the resolved artifact and Ollama version. Begin with text-only prompts and bounded context. If Ollama reports CPU offload or the system begins swapping, the model may technically run while behaving very differently from a full accelerator-resident deployment.

Vision is a separate validation step

The community repository lists a separate 0.928GB F16 vision projector. A successful text chat does not prove that images work or that the complete multimodal path fits. Verify that your runtime loads the matching projector and record image resolution, image count, preprocessing time, peak memory, and output correctness separately.

Do not mix a projector from another model or revision. If the runtime’s automatic vision path is unclear, keep the result labeled text-only until the loaded files are visible in logs or runtime diagnostics.

MTP and reasoning controls are not free defaults

The repository also lists a separate 1.370GB Q4_0 MTP draft file. Only enable speculative or MTP behavior when your runtime documents support for this exact pairing. Record whether the draft file was loaded, the speculative settings, accepted-token behavior, and complete memory cost.

Qwen describes thinking mode as enabled by default and supports reasoning controls. Preserve the prompt template and reasoning settings in comparisons; changing them can change output length, latency, and apparent throughput.

Official-checkpoint server path

If you intend to use Qwen’s official standard repository rather than a GGUF conversion, the project documents this Transformers server command:

transformers serve Qwen/Qwen3.8-27B --port 8000 --continuous-batching

Qwen also documents SGLang, vLLM, and TokenSpeed deployment examples. Their published examples use tensor parallelism across four devices and a 262,144-token context setting. Reproduce those named parameters only on hardware and software that support them; do not remove parallelism flags and assume the result is equivalent.

The official FP8 and standard checkpoints require materially more memory than UD-Q4_K_M. Confirm the exact repository before applying the capacity guidance.

A benchmark record that other readers can evaluate

Capture all of the following:

• machine, CPU, accelerator model and count, per-device memory, system memory, operating system, and driver;
• runtime name, version or commit, backend, command line, and device placement;
• repository, exact revision, filename, checksum, quantization, projector, and MTP file;
• context allocation, prompt tokens, generated tokens, cache precision, concurrency, and modality;
• cold-load time, time to first token, prompt-processing rate, generation rate, peak memory, fallback/offload, and errors;
• the prompt or workload and enough output evidence to assess correctness.

A model opening successfully is not a benchmark. Estimated speed, a one-line anecdote, or an unlabeled offloaded result should not be presented as a measured full-GPU result.

Continue from the successful smoke test

• Stage a reproducible benchmark submission
• Estimate a complete hardware budget
• Browse screened high-VRAM hardware observations
• Save a private browser-only hardware watchlist
• Ask a question or report a correction

Primary sources

• Official Qwen3.8 repository and local-use guidance
• Official Qwen3.8-27B model card
• Unsloth community GGUF file tree and current commands
• llama.cpp official repository
• HighVRAM methodology

Bottom line: use UD-Q4_K_M on a 24GB-or-larger system for the cleanest consumer starting point, validate text first, and add context, vision, MTP, or concurrency one variable at a time. Use Qwen’s official checkpoint paths only when your runtime and hardware match those larger artifacts.