Browse cited local AI measurements from original testers and project maintainers. HighVRAM has not independently run or reproduced these tests. Different models, runtime builds, workload shapes and timing definitions prevent an overall hardware ranking.
Contribute a reproducible record
Use the benchmark submission builder to validate the minimum evidence and download a structured JSON record. The builder uploads nothing and clearly separates unreviewed, source-checked and reproduced results.
Evidence wanted: next-generation accelerators and controlled RTX pairs
The current fixed snapshot contains no accepted observations for B200, B300, MI350X, MI355X, RTX PRO 6000 Blackwell, RTX 6000 Ada, RTX A6000, A40, L40S, A100 or H100. That absence is not a performance, compatibility or availability claim. A useful first record should include the exact system or board, memory exposed per accelerator, driver and CUDA or ROCm version, framework/container, model revision, precision, context, concurrency, topology, placement, offload, measured peak memory, timing definition, repetitions and public source.
The snapshot does contain four RTX 5090 metric observations from one source profile series, but the accelerator count is unreported and there is no controlled same-workload RTX PRO 6000 pair. A useful comparison must hold model artifact, runtime, precision, context, batch/concurrency and timing definition constant, identify the exact RTX PRO 6000 edition, and report peak memory as well as performance.
There are also no accepted controlled RTX A6000–RTX 6000 Ada, RTX 6000 Ada–H100, RTX A6000–H100, RTX A6000–A100, RTX 6000 Ada–A100, L40S–A100, L40S–H100 or A40–A100 pairs. A useful record must identify the exact 48GB workstation card and, when applicable, H100 PCIe 80GB, SXM 80GB or NVL 94GB; report the complete host, cooling, power mode, NVLink/MIG or other topology; hold the model artifact, runtime, precision, context, concurrency and timing boundary constant; and report exposed and peak memory separately.
Browse metric observations
Filter by hardware report and metric. Matching source tables appear directly below; definitions and settings stay attached. This fixed snapshot contains 43 observations, not an overall hardware ranking.
Results follow below. Jump to interpretation notes, the source index or downloadable data.
No observations in this snapshot match both selections. Try another metric or use Reset to show all observations. A missing result is not a performance or compatibility claim.
M3 Ultra: Rapid-MLX streaming requests
Source published: September 2, 2026. Exact execution date: not reported. Read the pinned original benchmark report. Accessed October 8, 2026.
Source type: Project-owner measurements from raullenchai / Rapid-MLX. Mac Studio Mac15,14 with Apple M3 Ultra, 28 CPU cores and 256 GB unified memory; macOS 26.5.2. The GPU-core count and selected backend are not reported.
Model and runtime: Qwen3.8-Flash-Next using the source’s 4-bit alias; ordinary autoregressive decoding, with speculative/MTP experiments excluded. Rapid-MLX 0.13.4 candidate, MLX 0.32.2, MLX-LM 0.31.3 and MLX-VLM 0.6.17. The exact reported runtime commit and artifact revision remain in the record details and download.
Request setup: Batch size 1, one resident model, temperature 0 and thinking disabled. Prefix cache is cleared before each timed request. Each reported value is a median of three requests after server readiness. Configured context capacity and concurrency are not explicitly recorded.
Important metric definitions: “Prompt/TTFT proxy” means server-reported prompt tokens divided by streamed time to first token. It includes serving/request overhead and is not an isolated engine-prefill rate. Streamed TTFT ends at the first visible content, reasoning or tool delta.
Decode correction: Published decode medians use (completion tokens − 1) / (total time − TTFT). The committed run CSV initially counts all completion tokens; the source metadata and source correction note specify the first-token correction. The displayed medians match that correction at the source’s published precision. Arithmetic checks do not reproduce the hardware measurements.
Read actual lengths: Prompt targets 128 / 2,048 / 8,192 / 32,768 correspond to server-reported counts 92 / 2,012 / 8,156 / 32,732. The shortest profile generated 232 tokens; the other profiles generated 256. These lengths do not establish configured context capacity.
Memory limitation: After-sweep allocator readings of 102.78 GB active and 148.08 GB peak include a peak inherited from model loading. They are not per-profile memory measurements; peak-memory fields remain unknown. Measured power is not reported.
Comparison limit: This source uses its own model, runtime, workload and timing definitions. It does not establish a controlled comparison with the other hardware on this page.
Prompt/TTFT proxy (tokens/s)
| Reported tokens | Reported tokens/s | Source & details |
|---|---|---|
| 92 prompt 232 generated | 262.0 | Original Details |
| 2,012 prompt 256 generated | 875.1 | Original Details |
| 8,156 prompt 256 generated | 867.9 | Original Details |
| 32,732 prompt 256 generated | 715.8 | Original Details |
Corrected decode (tokens/s)
| Reported tokens | Reported tokens/s | Source & details |
|---|---|---|
| 92 prompt 232 generated | 25.35 | Original Details |
| 2,012 prompt 256 generated | 23.98 | Original Details |
| 8,156 prompt 256 generated | 23.03 | Original Details |
| 32,732 prompt 256 generated | 21.37 | Original Details |
Streamed time to first token (ms)
| Reported tokens | Reported ms | Source & details |
|---|---|---|
| 92 prompt 232 generated | 351 | Original Details |
| 2,012 prompt 256 generated | 2,299 | Original Details |
| 8,156 prompt 256 generated | 9,397 | Original Details |
| 32,732 prompt 256 generated | 45,731 | Original Details |
Configuration and record details
92 prompt / 232 generated tokens
Profile identifier: profile-e94bc07f718dc62e. Metric records in this profile describe one reported configuration group; they are not independent runs.
Source locator: docs/benchmarks/recent-large-models-m3-ultra.md L166; Qwen3.8-Flash-Next context curve, target 128 / reported 92; run CSV lines 38,39,40. Open the pinned original report.
- Hardware: Apple M3 Ultra (Mac Studio Mac15,14)
- Reported system memory: 256 GB; memory type: unified. This is a host capacity, not measured per-model memory use.
- Model / quantization: Qwen3.8-Flash-Next / 4-bit (Rapid-MLX alias; detailed tensor/group policy unreported in benchmark)
- Artifact label: rapid-mlx/Qwen3.8-Flash-Next-4bit @ dcf657e4acda2aae72da99cde65b6c491cd96998
- Runtime: Rapid-MLX (vllm_mlx package) / MLX; 0.13.4 candidate; commit 615a8c5cd17b40db8d49e17d93c96f9094f23221; MLX 0.32.2; MLX-LM 0.31.3; MLX-VLM 0.6.17; selected backend: Not reported
- Workload: Cold-prefix-cache streaming request; target prompt 128, server-reported prompt 92; 256 requested output tokens, 232 actual. Ordinary autoregressive path, thinking disabled, temperature zero.
- Batch size / concurrency / configured context: 1 / Not reported / Not reported
- Prompt/TTFT proxy: 262.0 tokens/s. Record: rapid-20260902-m3ultra-qwen38-flashnext-q4-p92-prefill.
- Corrected decode: 25.35 tokens/s. Record: rapid-20260902-m3ultra-qwen38-flashnext-q4-p92-decode.
- Streamed time to first token: 351 ms. Record: rapid-20260902-m3ultra-qwen38-flashnext-q4-p92-ttft.
The downloadable JSON preserves all source notes and unknown fields.
2,012 prompt / 256 generated tokens
Profile identifier: profile-81dea56fb4825f18. Metric records in this profile describe one reported configuration group; they are not independent runs.
Source locator: docs/benchmarks/recent-large-models-m3-ultra.md L167; Qwen3.8-Flash-Next context curve, target 2048 / reported 2012; run CSV lines 41,42,43. Open the pinned original report.
- Hardware: Apple M3 Ultra (Mac Studio Mac15,14)
- Reported system memory: 256 GB; memory type: unified. This is a host capacity, not measured per-model memory use.
- Model / quantization: Qwen3.8-Flash-Next / 4-bit (Rapid-MLX alias; detailed tensor/group policy unreported in benchmark)
- Artifact label: rapid-mlx/Qwen3.8-Flash-Next-4bit @ dcf657e4acda2aae72da99cde65b6c491cd96998
- Runtime: Rapid-MLX (vllm_mlx package) / MLX; 0.13.4 candidate; commit 615a8c5cd17b40db8d49e17d93c96f9094f23221; MLX 0.32.2; MLX-LM 0.31.3; MLX-VLM 0.6.17; selected backend: Not reported
- Workload: Cold-prefix-cache streaming request; target prompt 2048, server-reported prompt 2012; 256 requested output tokens, 256 actual. Ordinary autoregressive path, thinking disabled, temperature zero.
- Batch size / concurrency / configured context: 1 / Not reported / Not reported
- Prompt/TTFT proxy: 875.1 tokens/s. Record: rapid-20260902-m3ultra-qwen38-flashnext-q4-p2012-prefill.
- Corrected decode: 23.98 tokens/s. Record: rapid-20260902-m3ultra-qwen38-flashnext-q4-p2012-decode.
- Streamed time to first token: 2,299 ms. Record: rapid-20260902-m3ultra-qwen38-flashnext-q4-p2012-ttft.
The downloadable JSON preserves all source notes and unknown fields.
8,156 prompt / 256 generated tokens
Profile identifier: profile-d65bc9dbf8ac56e1. Metric records in this profile describe one reported configuration group; they are not independent runs.
Source locator: docs/benchmarks/recent-large-models-m3-ultra.md L168; Qwen3.8-Flash-Next context curve, target 8192 / reported 8156; run CSV lines 44,45,46. Open the pinned original report.
- Hardware: Apple M3 Ultra (Mac Studio Mac15,14)
- Reported system memory: 256 GB; memory type: unified. This is a host capacity, not measured per-model memory use.
- Model / quantization: Qwen3.8-Flash-Next / 4-bit (Rapid-MLX alias; detailed tensor/group policy unreported in benchmark)
- Artifact label: rapid-mlx/Qwen3.8-Flash-Next-4bit @ dcf657e4acda2aae72da99cde65b6c491cd96998
- Runtime: Rapid-MLX (vllm_mlx package) / MLX; 0.13.4 candidate; commit 615a8c5cd17b40db8d49e17d93c96f9094f23221; MLX 0.32.2; MLX-LM 0.31.3; MLX-VLM 0.6.17; selected backend: Not reported
- Workload: Cold-prefix-cache streaming request; target prompt 8192, server-reported prompt 8156; 256 requested output tokens, 256 actual. Ordinary autoregressive path, thinking disabled, temperature zero.
- Batch size / concurrency / configured context: 1 / Not reported / Not reported
- Prompt/TTFT proxy: 867.9 tokens/s. Record: rapid-20260902-m3ultra-qwen38-flashnext-q4-p8156-prefill.
- Corrected decode: 23.03 tokens/s. Record: rapid-20260902-m3ultra-qwen38-flashnext-q4-p8156-decode.
- Streamed time to first token: 9,397 ms. Record: rapid-20260902-m3ultra-qwen38-flashnext-q4-p8156-ttft.
The downloadable JSON preserves all source notes and unknown fields.
32,732 prompt / 256 generated tokens
Profile identifier: profile-d607d83aea8a54d4. Metric records in this profile describe one reported configuration group; they are not independent runs.
Source locator: docs/benchmarks/recent-large-models-m3-ultra.md L169; Qwen3.8-Flash-Next context curve, target 32768 / reported 32732; run CSV lines 47,48,49. Open the pinned original report.
- Hardware: Apple M3 Ultra (Mac Studio Mac15,14)
- Reported system memory: 256 GB; memory type: unified. This is a host capacity, not measured per-model memory use.
- Model / quantization: Qwen3.8-Flash-Next / 4-bit (Rapid-MLX alias; detailed tensor/group policy unreported in benchmark)
- Artifact label: rapid-mlx/Qwen3.8-Flash-Next-4bit @ dcf657e4acda2aae72da99cde65b6c491cd96998
- Runtime: Rapid-MLX (vllm_mlx package) / MLX; 0.13.4 candidate; commit 615a8c5cd17b40db8d49e17d93c96f9094f23221; MLX 0.32.2; MLX-LM 0.31.3; MLX-VLM 0.6.17; selected backend: Not reported
- Workload: Cold-prefix-cache streaming request; target prompt 32768, server-reported prompt 32732; 256 requested output tokens, 256 actual. Ordinary autoregressive path, thinking disabled, temperature zero.
- Batch size / concurrency / configured context: 1 / Not reported / Not reported
- Prompt/TTFT proxy: 715.8 tokens/s. Record: rapid-20260902-m3ultra-qwen38-flashnext-q4-p32732-prefill.
- Corrected decode: 21.37 tokens/s. Record: rapid-20260902-m3ultra-qwen38-flashnext-q4-p32732-decode.
- Streamed time to first token: 45,731 ms. Record: rapid-20260902-m3ultra-qwen38-flashnext-q4-p32732-ttft.
The downloadable JSON preserves all source notes and unknown fields.
M4 Max: official MLX-LM benchmarks
Source published: October 15, 2025. Exact execution date: not reported. Read the pinned original benchmark report. Accessed October 8, 2026.
Source type: Maintainer-authored measurements by Awni Hannun in the official MLX-LM repository. The reported host is Apple M4 Max with 64 GB memory and macOS 26.1; exact device model, core counts and selected backend are not stated.
Reported runtime and workload: MLX-LM 0.28.2 with MLX 0.29.2.dev20251008+85a8824a8. The command specifies 2,048 prompt tokens and 128 generated tokens. The dependency-version date is not an execution timestamp. Prompt and generation rates are separate published columns.
Model distinction: These q4 rows use Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507. The source’s q4 label is not q4 g32 or a GGUF quantization label. Artifact repositories are reported; exact measured revisions and file hashes are not.
Not reported: Batch size, concurrency, configured context, trial count, aggregation, cache state, timing boundaries, TTFT and measured power. No settings are inferred from software defaults. The generic “Memory GB” column has no defined measurement boundary, so it is not published here as peak memory.
Comparison limit: This source uses its own model, runtime, workload and timing definitions. It does not establish a controlled comparison with the other hardware on this page.
Prompt processing (tokens/s)
| Model and workload | Reported tokens/s | Source & details |
|---|---|---|
| Qwen3-4B-Instruct-2507 q4; 2,048 prompt / 128 generated | 1,622.27 | Original Details |
| Qwen3-30B-A3B-Instruct-2507 q4; 2,048 prompt / 128 generated | 1,753.90 | Original Details |
Token generation (tokens/s)
| Model and workload | Reported tokens/s | Source & details |
|---|---|---|
| Qwen3-4B-Instruct-2507 q4; 2,048 prompt / 128 generated | 134.52 | Original Details |
| Qwen3-30B-A3B-Instruct-2507 q4; 2,048 prompt / 128 generated | 113.33 | Original Details |
Configuration and record details
Qwen3-30B-A3B-Instruct-2507 · q4
Profile identifier: profile-34197ba17e85d650. Metric records in this profile describe one reported configuration group; they are not independent runs.
Source locator: mlx_lm/BENCHMARKS.md L55; Qwen3-30B-A3B-Instruct-2507, q4 row; environment L58-L61; command L14. Open the pinned original report.
- Hardware: Apple M4 Max
- Reported system memory: 64 GB; memory type: Not reported. This is a host capacity, not measured per-model memory use.
- Model / quantization: Qwen3-30B-A3B-Instruct-2507 / q4 (source label; 4-bit artifact)
- Artifact label: mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit
- Runtime: MLX-LM / mlx_lm.benchmark; mlx-lm 0.28.2; mlx 0.29.2.dev20251008+85a8824a8; selected backend: Not reported
- Workload: Source efficiency benchmark command specifies prompt and generation lengths; phase rates are separate published columns.
- Batch size / concurrency / configured context: Not reported / Not reported / Not reported
- Prompt processing: 1,753.90 tokens/s. Record: awni-20251015-m4max-qwen3-30b-a3b-q4-prompt2048.
- Token generation: 113.33 tokens/s. Record: awni-20251015-m4max-qwen3-30b-a3b-q4-generation128.
The source Memory GB column is 18.20; its measurement boundary is not defined in the table, so peak_memory_gb is null.
The downloadable JSON preserves all source notes and unknown fields.
Qwen3-4B-Instruct-2507 · q4
Profile identifier: profile-cf799631f7ce6285. Metric records in this profile describe one reported configuration group; they are not independent runs.
Source locator: mlx_lm/BENCHMARKS.md L37; Qwen3-4B-Instruct-2507, q4 row; environment L39-L42; command L14. Open the pinned original report.
- Hardware: Apple M4 Max
- Reported system memory: 64 GB; memory type: Not reported. This is a host capacity, not measured per-model memory use.
- Model / quantization: Qwen3-4B-Instruct-2507 / q4 (source label; 4-bit artifact)
- Artifact label: mlx-community/Qwen3-4B-Instruct-2507-4bit
- Runtime: MLX-LM / mlx_lm.benchmark; mlx-lm 0.28.2; mlx 0.29.2.dev20251008+85a8824a8; selected backend: Not reported
- Workload: Source efficiency benchmark command specifies prompt and generation lengths; phase rates are separate published columns.
- Batch size / concurrency / configured context: Not reported / Not reported / Not reported
- Prompt processing: 1,622.27 tokens/s. Record: awni-20251015-m4max-qwen3-4b-q4-prompt2048.
- Token generation: 134.52 tokens/s. Record: awni-20251015-m4max-qwen3-4b-q4-generation128.
The source Memory GB column is 3.35; its measurement boundary is not defined in the table, so peak_memory_gb is null.
The downloadable JSON preserves all source notes and unknown fields.
RTX 4090: lhl
Source published: August 6, 2025. Exact execution date: not reported. Read lhl’s original benchmark post. Accessed October 8, 2026.
Reported configuration: Llama 2 7B; quantization Q4_0; llama.cpp / llama-bench; build 224145325 (build 6098); backend CUDA. Exact artifact hashes are not reported. Model-version labels follow the individual source; matching parameter counts and quantization names do not establish that two reports used the same model.
Not reported: batch size, context capacity, prior context depth, concurrency, repetition count, measured power, time to first token and peak memory. No memory capacity is assumed from the product name.
- Source reports one visible CUDA device, compute capability 8.9 and VMM enabled.
- Reported CUDA toolkit: 12.9 / V12.9.86; NVIDIA driver package: nvidia-dkms 575.64.03-1.
- Board vendor, CPU, RAM, VRAM capacity, operating system and power limit are not explicitly reported for this run.
Prompt processing: pp512
| Configuration | Reported tokens/s | Source & settings |
|---|---|---|
| RTX 4090 Flash attention off | 11,992.70 ±107.99 | Original Details |
| RTX 4090 Flash attention on | 14,770.63 ±102.93 | Original Details |
Token generation: tg128
| Configuration | Reported tokens/s | Source & settings |
|---|---|---|
| RTX 4090 Flash attention off | 186.21 ±0.13 | Original Details |
| RTX 4090 Flash attention on | 188.96 ±0.05 | Original Details |
Record details
RTX 4090 · pp512 · Flash attention off
Record: lhl-20250806-rtx4090-fa0-pp512
Profile identifier: profile-abc8455a0e2ee64b. Profile IDs preserve reported configuration and provenance; they do not certify comparability.
Prompt processing: 11,992.70 tokens/s.
Source locator: discussioncomment-14016393; Non-FA; pp512 row.
Open the cited original post.
- Hardware: NVIDIA GeForce RTX 4090
- Accelerator count: 1
- Accelerator memory / system memory: Not reported / Not reported
- Model filename/artifact label: llama-2-7b.Q4_0.gguf
- Model / quantization: Llama 2 7B / Q4_0
- Runtime / backend: llama.cpp / llama-bench 224145325 (build 6098) / CUDA
- Test description: llama-bench separate pp512 test; flash attention disabled
- Prompt tokens / generated tokens: 512 / Not reported
- Batch size / context setting / concurrency: Not reported / Not reported / Not reported
Source-specific notes:
- Externally reported by the original tester; not measured or reproduced by HighVRAM Hardware Watch.
- Publication date is the comment date. Exact benchmark execution date is unknown.
- Prompt processing and generation are separate microbenchmark tests; pp512 is not the prompt paired with tg128.
- Context capacity, prior context depth, concurrency and repetition count are not reported. Do not infer them from the test label.
- The command explicitly uses llama-2-7b.Q4_0.gguf. Exact artifact hash is unreported.
- Reported model size: 3.56 GiB; parameters: 6.74 B. Model size is not peak memory.
- GPU layer setting (ngl): 99.
- Reported dispersion: ±107.99 tokens/s; the comment does not define its statistical interpretation.
- Different runtime commits, flash-attention settings, power limits and undisclosed host settings prevent a universal cross-source hardware ranking.
- Source reports one visible CUDA device, compute capability 8.9 and VMM enabled.
- Reported CUDA toolkit: 12.9 / V12.9.86; NVIDIA driver package: nvidia-dkms 575.64.03-1.
- GGML_CUDA_FORCE_MMQ and GGML_CUDA_FORCE_CUBLAS are both reported as no.
- Board vendor, CPU, RAM, VRAM capacity, operating system and power limit are not explicitly reported for this run.
RTX 4090 · tg128 · Flash attention off
Record: lhl-20250806-rtx4090-fa0-tg128
Profile identifier: profile-b95e3df9708a10e8. Profile IDs preserve reported configuration and provenance; they do not certify comparability.
Token generation: 186.21 tokens/s.
Source locator: discussioncomment-14016393; Non-FA; tg128 row.
Open the cited original post.
- Hardware: NVIDIA GeForce RTX 4090
- Accelerator count: 1
- Accelerator memory / system memory: Not reported / Not reported
- Model filename/artifact label: llama-2-7b.Q4_0.gguf
- Model / quantization: Llama 2 7B / Q4_0
- Runtime / backend: llama.cpp / llama-bench 224145325 (build 6098) / CUDA
- Test description: llama-bench separate tg128 test; flash attention disabled
- Prompt tokens / generated tokens: Not reported / 128
- Batch size / context setting / concurrency: Not reported / Not reported / Not reported
Source-specific notes:
- Externally reported by the original tester; not measured or reproduced by HighVRAM Hardware Watch.
- Publication date is the comment date. Exact benchmark execution date is unknown.
- Prompt processing and generation are separate microbenchmark tests; pp512 is not the prompt paired with tg128.
- Context capacity, prior context depth, concurrency and repetition count are not reported. Do not infer them from the test label.
- The command explicitly uses llama-2-7b.Q4_0.gguf. Exact artifact hash is unreported.
- Reported model size: 3.56 GiB; parameters: 6.74 B. Model size is not peak memory.
- GPU layer setting (ngl): 99.
- Reported dispersion: ±0.13 tokens/s; the comment does not define its statistical interpretation.
- Different runtime commits, flash-attention settings, power limits and undisclosed host settings prevent a universal cross-source hardware ranking.
- Source reports one visible CUDA device, compute capability 8.9 and VMM enabled.
- Reported CUDA toolkit: 12.9 / V12.9.86; NVIDIA driver package: nvidia-dkms 575.64.03-1.
- GGML_CUDA_FORCE_MMQ and GGML_CUDA_FORCE_CUBLAS are both reported as no.
- Board vendor, CPU, RAM, VRAM capacity, operating system and power limit are not explicitly reported for this run.
RTX 4090 · pp512 · Flash attention on
Record: lhl-20250806-rtx4090-fa1-pp512
Profile identifier: profile-b67124fb1dca9609. Profile IDs preserve reported configuration and provenance; they do not certify comparability.
Prompt processing: 14,770.63 tokens/s.
Source locator: discussioncomment-14016393; FA; pp512 row.
Open the cited original post.
- Hardware: NVIDIA GeForce RTX 4090
- Accelerator count: 1
- Accelerator memory / system memory: Not reported / Not reported
- Model filename/artifact label: llama-2-7b.Q4_0.gguf
- Model / quantization: Llama 2 7B / Q4_0
- Runtime / backend: llama.cpp / llama-bench 224145325 (build 6098) / CUDA
- Test description: llama-bench separate pp512 test; flash attention enabled
- Prompt tokens / generated tokens: 512 / Not reported
- Batch size / context setting / concurrency: Not reported / Not reported / Not reported
Source-specific notes:
- Externally reported by the original tester; not measured or reproduced by HighVRAM Hardware Watch.
- Publication date is the comment date. Exact benchmark execution date is unknown.
- Prompt processing and generation are separate microbenchmark tests; pp512 is not the prompt paired with tg128.
- Context capacity, prior context depth, concurrency and repetition count are not reported. Do not infer them from the test label.
- The command explicitly uses llama-2-7b.Q4_0.gguf. Exact artifact hash is unreported.
- Reported model size: 3.56 GiB; parameters: 6.74 B. Model size is not peak memory.
- GPU layer setting (ngl): 99.
- Reported dispersion: ±102.93 tokens/s; the comment does not define its statistical interpretation.
- Different runtime commits, flash-attention settings, power limits and undisclosed host settings prevent a universal cross-source hardware ranking.
- Source reports one visible CUDA device, compute capability 8.9 and VMM enabled.
- Reported CUDA toolkit: 12.9 / V12.9.86; NVIDIA driver package: nvidia-dkms 575.64.03-1.
- GGML_CUDA_FORCE_MMQ and GGML_CUDA_FORCE_CUBLAS are both reported as no.
- Board vendor, CPU, RAM, VRAM capacity, operating system and power limit are not explicitly reported for this run.
RTX 4090 · tg128 · Flash attention on
Record: lhl-20250806-rtx4090-fa1-tg128
Profile identifier: profile-a5fe42ff71abce7c. Profile IDs preserve reported configuration and provenance; they do not certify comparability.
Token generation: 188.96 tokens/s.
Source locator: discussioncomment-14016393; FA; tg128 row.
Open the cited original post.
- Hardware: NVIDIA GeForce RTX 4090
- Accelerator count: 1
- Accelerator memory / system memory: Not reported / Not reported
- Model filename/artifact label: llama-2-7b.Q4_0.gguf
- Model / quantization: Llama 2 7B / Q4_0
- Runtime / backend: llama.cpp / llama-bench 224145325 (build 6098) / CUDA
- Test description: llama-bench separate tg128 test; flash attention enabled
- Prompt tokens / generated tokens: Not reported / 128
- Batch size / context setting / concurrency: Not reported / Not reported / Not reported
Source-specific notes:
- Externally reported by the original tester; not measured or reproduced by HighVRAM Hardware Watch.
- Publication date is the comment date. Exact benchmark execution date is unknown.
- Prompt processing and generation are separate microbenchmark tests; pp512 is not the prompt paired with tg128.
- Context capacity, prior context depth, concurrency and repetition count are not reported. Do not infer them from the test label.
- The command explicitly uses llama-2-7b.Q4_0.gguf. Exact artifact hash is unreported.
- Reported model size: 3.56 GiB; parameters: 6.74 B. Model size is not peak memory.
- GPU layer setting (ngl): 99.
- Reported dispersion: ±0.05 tokens/s; the comment does not define its statistical interpretation.
- Different runtime commits, flash-attention settings, power limits and undisclosed host settings prevent a universal cross-source hardware ranking.
- Source reports one visible CUDA device, compute capability 8.9 and VMM enabled.
- Reported CUDA toolkit: 12.9 / V12.9.86; NVIDIA driver package: nvidia-dkms 575.64.03-1.
- GGML_CUDA_FORCE_MMQ and GGML_CUDA_FORCE_CUBLAS are both reported as no.
- Board vendor, CPU, RAM, VRAM capacity, operating system and power limit are not explicitly reported for this run.
RTX 3090, RTX 5090: RodriMora
Source published: August 2, 2025. Exact execution date: not reported. Read RodriMora’s original benchmark post. Accessed October 8, 2026.
Reported configuration: Llama 7B (source label; version unverified); quantization Q4_0; llama.cpp / llama-bench; build 9c35706 (build 6060); backend CUDA. Exact artifact hashes are not reported. Model-version labels follow the individual source; matching parameter counts and quantization names do not establish that two reports used the same model.
Not reported: batch size, context capacity, prior context depth, concurrency, repetition count, measured power, time to first token and peak memory. No memory capacity is assumed from the product name.
- RTX 3090: Reported GPU power limit: 390 W; this is not measured power consumption.
- GPU count, board vendor, CPU, host RAM, VRAM capacity, operating system and CUDA/driver versions are not explicitly reported for this run.
- RTX 5090: Reported GPU power limit: 600 W; this is not measured power consumption.
Prompt processing: pp512
| Configuration | Reported tokens/s | Source & settings |
|---|---|---|
| RTX 3090 Flash attention off | 5,405.83 ±5.80 | Original Details |
| RTX 3090 Flash attention on | 5,932.44 ±10.87 | Original Details |
| RTX 5090 Flash attention off | 14,751.98 ±136.24 | Original Details |
| RTX 5090 Flash attention on | 16,041.54 ±85.27 | Original Details |
Token generation: tg128
| Configuration | Reported tokens/s | Source & settings |
|---|---|---|
| RTX 3090 Flash attention off | 151.04 ±0.24 | Original Details |
| RTX 3090 Flash attention on | 155.36 ±0.09 | Original Details |
| RTX 5090 Flash attention off | 239.62 ±0.37 | Original Details |
| RTX 5090 Flash attention on | 248.57 ±0.05 | Original Details |
Record details
RTX 3090 · pp512 · Flash attention off
Record: rodri-20250802-rtx3090-fa0-pp512
Profile identifier: profile-077891c574cb5dfc. Profile IDs preserve reported configuration and provenance; they do not certify comparability.
Prompt processing: 5,405.83 tokens/s.
Source locator: discussioncomment-13968120; RTX 3090 at 390 W; fa=0; pp512 row.
Open the cited original post.
- Hardware: NVIDIA GeForce RTX 3090
- Accelerator count: Not reported
- Accelerator memory / system memory: Not reported / Not reported
- Model filename/artifact label: Not reported
- Model / quantization: Llama 7B (source label; version unverified) / Q4_0
- Runtime / backend: llama.cpp / llama-bench 9c35706 (build 6060) / CUDA
- Test description: llama-bench separate pp512 test; flash attention disabled
- Prompt tokens / generated tokens: 512 / Not reported
- Batch size / context setting / concurrency: Not reported / Not reported / Not reported
Source-specific notes:
- Externally reported by the original tester; not measured or reproduced by HighVRAM Hardware Watch.
- Publication date is the comment date. Exact benchmark execution date is unknown.
- Prompt processing and generation are separate microbenchmark tests; pp512 is not the prompt paired with tg128.
- Context capacity, prior context depth, concurrency and repetition count are not reported. Do not infer them from the test label.
- The exact source row identifies llama 7B. Model version and artifact are unverified; the surrounding discussion protocol does not establish them for this run.
- Reported model size: 3.56 GiB; parameters: 6.74 B. Model size is not peak memory.
- GPU layer setting (ngl): 99.
- Reported dispersion: ±5.80 tokens/s; the comment does not define its statistical interpretation.
- Different runtime commits, flash-attention settings, power limits and undisclosed host settings prevent a universal cross-source hardware ranking.
- Reported GPU power limit: 390 W; this is not measured power consumption.
- GPU count, board vendor, CPU, host RAM, VRAM capacity, operating system and CUDA/driver versions are not explicitly reported for this run.
RTX 3090 · tg128 · Flash attention off
Record: rodri-20250802-rtx3090-fa0-tg128
Profile identifier: profile-e92fff03736421b6. Profile IDs preserve reported configuration and provenance; they do not certify comparability.
Token generation: 151.04 tokens/s.
Source locator: discussioncomment-13968120; RTX 3090 at 390 W; fa=0; tg128 row.
Open the cited original post.
- Hardware: NVIDIA GeForce RTX 3090
- Accelerator count: Not reported
- Accelerator memory / system memory: Not reported / Not reported
- Model filename/artifact label: Not reported
- Model / quantization: Llama 7B (source label; version unverified) / Q4_0
- Runtime / backend: llama.cpp / llama-bench 9c35706 (build 6060) / CUDA
- Test description: llama-bench separate tg128 test; flash attention disabled
- Prompt tokens / generated tokens: Not reported / 128
- Batch size / context setting / concurrency: Not reported / Not reported / Not reported
Source-specific notes:
- Externally reported by the original tester; not measured or reproduced by HighVRAM Hardware Watch.
- Publication date is the comment date. Exact benchmark execution date is unknown.
- Prompt processing and generation are separate microbenchmark tests; pp512 is not the prompt paired with tg128.
- Context capacity, prior context depth, concurrency and repetition count are not reported. Do not infer them from the test label.
- The exact source row identifies llama 7B. Model version and artifact are unverified; the surrounding discussion protocol does not establish them for this run.
- Reported model size: 3.56 GiB; parameters: 6.74 B. Model size is not peak memory.
- GPU layer setting (ngl): 99.
- Reported dispersion: ±0.24 tokens/s; the comment does not define its statistical interpretation.
- Different runtime commits, flash-attention settings, power limits and undisclosed host settings prevent a universal cross-source hardware ranking.
- Reported GPU power limit: 390 W; this is not measured power consumption.
- GPU count, board vendor, CPU, host RAM, VRAM capacity, operating system and CUDA/driver versions are not explicitly reported for this run.
RTX 3090 · pp512 · Flash attention on
Record: rodri-20250802-rtx3090-fa1-pp512
Profile identifier: profile-63df13801953bbcd. Profile IDs preserve reported configuration and provenance; they do not certify comparability.
Prompt processing: 5,932.44 tokens/s.
Source locator: discussioncomment-13968120; RTX 3090 at 390 W; fa=1; pp512 row.
Open the cited original post.
- Hardware: NVIDIA GeForce RTX 3090
- Accelerator count: Not reported
- Accelerator memory / system memory: Not reported / Not reported
- Model filename/artifact label: Not reported
- Model / quantization: Llama 7B (source label; version unverified) / Q4_0
- Runtime / backend: llama.cpp / llama-bench 9c35706 (build 6060) / CUDA
- Test description: llama-bench separate pp512 test; flash attention enabled
- Prompt tokens / generated tokens: 512 / Not reported
- Batch size / context setting / concurrency: Not reported / Not reported / Not reported
Source-specific notes:
- Externally reported by the original tester; not measured or reproduced by HighVRAM Hardware Watch.
- Publication date is the comment date. Exact benchmark execution date is unknown.
- Prompt processing and generation are separate microbenchmark tests; pp512 is not the prompt paired with tg128.
- Context capacity, prior context depth, concurrency and repetition count are not reported. Do not infer them from the test label.
- The exact source row identifies llama 7B. Model version and artifact are unverified; the surrounding discussion protocol does not establish them for this run.
- Reported model size: 3.56 GiB; parameters: 6.74 B. Model size is not peak memory.
- GPU layer setting (ngl): 99.
- Reported dispersion: ±10.87 tokens/s; the comment does not define its statistical interpretation.
- Different runtime commits, flash-attention settings, power limits and undisclosed host settings prevent a universal cross-source hardware ranking.
- Reported GPU power limit: 390 W; this is not measured power consumption.
- GPU count, board vendor, CPU, host RAM, VRAM capacity, operating system and CUDA/driver versions are not explicitly reported for this run.
RTX 3090 · tg128 · Flash attention on
Record: rodri-20250802-rtx3090-fa1-tg128
Profile identifier: profile-4f49ad4382f6b60d. Profile IDs preserve reported configuration and provenance; they do not certify comparability.
Token generation: 155.36 tokens/s.
Source locator: discussioncomment-13968120; RTX 3090 at 390 W; fa=1; tg128 row.
Open the cited original post.
- Hardware: NVIDIA GeForce RTX 3090
- Accelerator count: Not reported
- Accelerator memory / system memory: Not reported / Not reported
- Model filename/artifact label: Not reported
- Model / quantization: Llama 7B (source label; version unverified) / Q4_0
- Runtime / backend: llama.cpp / llama-bench 9c35706 (build 6060) / CUDA
- Test description: llama-bench separate tg128 test; flash attention enabled
- Prompt tokens / generated tokens: Not reported / 128
- Batch size / context setting / concurrency: Not reported / Not reported / Not reported
Source-specific notes:
- Externally reported by the original tester; not measured or reproduced by HighVRAM Hardware Watch.
- Publication date is the comment date. Exact benchmark execution date is unknown.
- Prompt processing and generation are separate microbenchmark tests; pp512 is not the prompt paired with tg128.
- Context capacity, prior context depth, concurrency and repetition count are not reported. Do not infer them from the test label.
- The exact source row identifies llama 7B. Model version and artifact are unverified; the surrounding discussion protocol does not establish them for this run.
- Reported model size: 3.56 GiB; parameters: 6.74 B. Model size is not peak memory.
- GPU layer setting (ngl): 99.
- Reported dispersion: ±0.09 tokens/s; the comment does not define its statistical interpretation.
- Different runtime commits, flash-attention settings, power limits and undisclosed host settings prevent a universal cross-source hardware ranking.
- Reported GPU power limit: 390 W; this is not measured power consumption.
- GPU count, board vendor, CPU, host RAM, VRAM capacity, operating system and CUDA/driver versions are not explicitly reported for this run.
RTX 5090 · pp512 · Flash attention off
Record: rodri-20250802-rtx5090-fa0-pp512
Profile identifier: profile-efad75c74af441f4. Profile IDs preserve reported configuration and provenance; they do not certify comparability.
Prompt processing: 14,751.98 tokens/s.
Source locator: discussioncomment-13968120; RTX 5090 at 600 W; fa=0; pp512 row.
Open the cited original post.
- Hardware: NVIDIA GeForce RTX 5090
- Accelerator count: Not reported
- Accelerator memory / system memory: Not reported / Not reported
- Model filename/artifact label: Not reported
- Model / quantization: Llama 7B (source label; version unverified) / Q4_0
- Runtime / backend: llama.cpp / llama-bench 9c35706 (build 6060) / CUDA
- Test description: llama-bench separate pp512 test; flash attention disabled
- Prompt tokens / generated tokens: 512 / Not reported
- Batch size / context setting / concurrency: Not reported / Not reported / Not reported
Source-specific notes:
- Externally reported by the original tester; not measured or reproduced by HighVRAM Hardware Watch.
- Publication date is the comment date. Exact benchmark execution date is unknown.
- Prompt processing and generation are separate microbenchmark tests; pp512 is not the prompt paired with tg128.
- Context capacity, prior context depth, concurrency and repetition count are not reported. Do not infer them from the test label.
- The exact source row identifies llama 7B. Model version and artifact are unverified; the surrounding discussion protocol does not establish them for this run.
- Reported model size: 3.56 GiB; parameters: 6.74 B. Model size is not peak memory.
- GPU layer setting (ngl): 99.
- Reported dispersion: ±136.24 tokens/s; the comment does not define its statistical interpretation.
- Different runtime commits, flash-attention settings, power limits and undisclosed host settings prevent a universal cross-source hardware ranking.
- Reported GPU power limit: 600 W; this is not measured power consumption.
- GPU count, board vendor, CPU, host RAM, VRAM capacity, operating system and CUDA/driver versions are not explicitly reported for this run.
RTX 5090 · tg128 · Flash attention off
Record: rodri-20250802-rtx5090-fa0-tg128
Profile identifier: profile-72dd09ca2082ddb8. Profile IDs preserve reported configuration and provenance; they do not certify comparability.
Token generation: 239.62 tokens/s.
Source locator: discussioncomment-13968120; RTX 5090 at 600 W; fa=0; tg128 row.
Open the cited original post.
- Hardware: NVIDIA GeForce RTX 5090
- Accelerator count: Not reported
- Accelerator memory / system memory: Not reported / Not reported
- Model filename/artifact label: Not reported
- Model / quantization: Llama 7B (source label; version unverified) / Q4_0
- Runtime / backend: llama.cpp / llama-bench 9c35706 (build 6060) / CUDA
- Test description: llama-bench separate tg128 test; flash attention disabled
- Prompt tokens / generated tokens: Not reported / 128
- Batch size / context setting / concurrency: Not reported / Not reported / Not reported
Source-specific notes:
- Externally reported by the original tester; not measured or reproduced by HighVRAM Hardware Watch.
- Publication date is the comment date. Exact benchmark execution date is unknown.
- Prompt processing and generation are separate microbenchmark tests; pp512 is not the prompt paired with tg128.
- Context capacity, prior context depth, concurrency and repetition count are not reported. Do not infer them from the test label.
- The exact source row identifies llama 7B. Model version and artifact are unverified; the surrounding discussion protocol does not establish them for this run.
- Reported model size: 3.56 GiB; parameters: 6.74 B. Model size is not peak memory.
- GPU layer setting (ngl): 99.
- Reported dispersion: ±0.37 tokens/s; the comment does not define its statistical interpretation.
- Different runtime commits, flash-attention settings, power limits and undisclosed host settings prevent a universal cross-source hardware ranking.
- Reported GPU power limit: 600 W; this is not measured power consumption.
- GPU count, board vendor, CPU, host RAM, VRAM capacity, operating system and CUDA/driver versions are not explicitly reported for this run.
RTX 5090 · pp512 · Flash attention on
Record: rodri-20250802-rtx5090-fa1-pp512
Profile identifier: profile-612432a9e5f7f71a. Profile IDs preserve reported configuration and provenance; they do not certify comparability.
Prompt processing: 16,041.54 tokens/s.
Source locator: discussioncomment-13968120; RTX 5090 at 600 W; fa=1; pp512 row.
Open the cited original post.
- Hardware: NVIDIA GeForce RTX 5090
- Accelerator count: Not reported
- Accelerator memory / system memory: Not reported / Not reported
- Model filename/artifact label: Not reported
- Model / quantization: Llama 7B (source label; version unverified) / Q4_0
- Runtime / backend: llama.cpp / llama-bench 9c35706 (build 6060) / CUDA
- Test description: llama-bench separate pp512 test; flash attention enabled
- Prompt tokens / generated tokens: 512 / Not reported
- Batch size / context setting / concurrency: Not reported / Not reported / Not reported
Source-specific notes:
- Externally reported by the original tester; not measured or reproduced by HighVRAM Hardware Watch.
- Publication date is the comment date. Exact benchmark execution date is unknown.
- Prompt processing and generation are separate microbenchmark tests; pp512 is not the prompt paired with tg128.
- Context capacity, prior context depth, concurrency and repetition count are not reported. Do not infer them from the test label.
- The exact source row identifies llama 7B. Model version and artifact are unverified; the surrounding discussion protocol does not establish them for this run.
- Reported model size: 3.56 GiB; parameters: 6.74 B. Model size is not peak memory.
- GPU layer setting (ngl): 99.
- Reported dispersion: ±85.27 tokens/s; the comment does not define its statistical interpretation.
- Different runtime commits, flash-attention settings, power limits and undisclosed host settings prevent a universal cross-source hardware ranking.
- Reported GPU power limit: 600 W; this is not measured power consumption.
- GPU count, board vendor, CPU, host RAM, VRAM capacity, operating system and CUDA/driver versions are not explicitly reported for this run.
RTX 5090 · tg128 · Flash attention on
Record: rodri-20250802-rtx5090-fa1-tg128
Profile identifier: profile-7f94a6577edf541e. Profile IDs preserve reported configuration and provenance; they do not certify comparability.
Token generation: 248.57 tokens/s.
Source locator: discussioncomment-13968120; RTX 5090 at 600 W; fa=1; tg128 row.
Open the cited original post.
- Hardware: NVIDIA GeForce RTX 5090
- Accelerator count: Not reported
- Accelerator memory / system memory: Not reported / Not reported
- Model filename/artifact label: Not reported
- Model / quantization: Llama 7B (source label; version unverified) / Q4_0
- Runtime / backend: llama.cpp / llama-bench 9c35706 (build 6060) / CUDA
- Test description: llama-bench separate tg128 test; flash attention enabled
- Prompt tokens / generated tokens: Not reported / 128
- Batch size / context setting / concurrency: Not reported / Not reported / Not reported
Source-specific notes:
- Externally reported by the original tester; not measured or reproduced by HighVRAM Hardware Watch.
- Publication date is the comment date. Exact benchmark execution date is unknown.
- Prompt processing and generation are separate microbenchmark tests; pp512 is not the prompt paired with tg128.
- Context capacity, prior context depth, concurrency and repetition count are not reported. Do not infer them from the test label.
- The exact source row identifies llama 7B. Model version and artifact are unverified; the surrounding discussion protocol does not establish them for this run.
- Reported model size: 3.56 GiB; parameters: 6.74 B. Model size is not peak memory.
- GPU layer setting (ngl): 99.
- Reported dispersion: ±0.05 tokens/s; the comment does not define its statistical interpretation.
- Different runtime commits, flash-attention settings, power limits and undisclosed host settings prevent a universal cross-source hardware ranking.
- Reported GPU power limit: 600 W; this is not measured power consumption.
- GPU count, board vendor, CPU, host RAM, VRAM capacity, operating system and CUDA/driver versions are not explicitly reported for this run.
M2 Max Mac Studio: brozkrut
Source published: November 23, 2023. Exact execution date: not reported. Read brozkrut’s original benchmark post. Accessed October 8, 2026.
Reported configuration: Llama 7B (source label; version unverified); quantization mostly Q4_0; llama.cpp / llama-bench; build 8e672ef (build 1550); backend Metal. Exact artifact hashes are not reported. Model-version labels follow the individual source; matching parameter counts and quantization names do not establish that two reports used the same model.
Not reported: batch size, context capacity, prior context depth, concurrency, repetition count, measured power, time to first token and peak memory. No memory capacity is assumed from the product name.
- The source specifies a Mac Studio with M2 Max, 8+4 CPU cores and 38 GPU cores. This 2023 Metal build predates the 2025 CUDA reports.
- Original quantization label: mostly Q4_0.
- Memory capacity, exact model artifact, operating-system version, flash-attention state and power limit are not reported in this post.
Prompt processing: pp512
| Configuration | Reported tokens/s | Source & settings |
|---|---|---|
| M2 Max Mac Studio Flash attention not reported | 671.31 ±0.20 | Original Details |
Token generation: tg128
| Configuration | Reported tokens/s | Source & settings |
|---|---|---|
| M2 Max Mac Studio Flash attention not reported | 65.95 ±0.08 | Original Details |
Record details
M2 Max Mac Studio · pp512 · Flash attention not reported
Record: brozkrut-20231123-m2max-pp512
Profile identifier: profile-9c142180994120ef. Profile IDs preserve reported configuration and provenance; they do not certify comparability.
Prompt processing: 671.31 tokens/s.
Source locator: discussioncomment-7653249; M2 Max Studio, 8+4 CPU, 38 GPU; mostly Q4_0; pp512 row.
Open the cited original post.
- Hardware: Apple M2 Max Mac Studio (8+4 CPU cores; 38 GPU cores)
- Accelerator count: Not reported
- Accelerator memory / system memory: Not reported / Not reported
- Model filename/artifact label: Not reported
- Model / quantization: Llama 7B (source label; version unverified) / mostly Q4_0
- Runtime / backend: llama.cpp / llama-bench 8e672ef (build 1550) / Metal
- Test description: llama-bench separate pp512 test; flash attention not reported
- Prompt tokens / generated tokens: 512 / Not reported
- Batch size / context setting / concurrency: Not reported / Not reported / Not reported
Source-specific notes:
- Externally reported by the original tester; not measured or reproduced by HighVRAM Hardware Watch.
- Publication date is the comment date. Exact benchmark execution date is unknown.
- Prompt processing and generation are separate microbenchmark tests; pp512 is not the prompt paired with tg128.
- Context capacity, prior context depth, concurrency and repetition count are not reported. Do not infer them from the test label.
- The exact source row identifies llama 7B. Model version and artifact are unverified; the surrounding discussion protocol does not establish them for this run.
- Reported model size: 3.56 GiB; parameters: 6.74 B. Model size is not peak memory.
- GPU layer setting (ngl): 99.
- Reported dispersion: ±0.20 tokens/s; the comment does not define its statistical interpretation.
- Different runtime commits, flash-attention settings, power limits and undisclosed host settings prevent a universal cross-source hardware ranking.
- Original quantization label: mostly Q4_0.
- Memory capacity, exact model artifact, operating-system version, flash-attention state and power limit are not reported in this post.
- Batch size is not stated in the original 2023 report. The subsequently updated thread methodology cannot establish this run setting; batch_size is null.
M2 Max Mac Studio · tg128 · Flash attention not reported
Record: brozkrut-20231123-m2max-tg128
Profile identifier: profile-91e1f81228754ab9. Profile IDs preserve reported configuration and provenance; they do not certify comparability.
Token generation: 65.95 tokens/s.
Source locator: discussioncomment-7653249; M2 Max Studio, 8+4 CPU, 38 GPU; mostly Q4_0; tg128 row.
Open the cited original post.
- Hardware: Apple M2 Max Mac Studio (8+4 CPU cores; 38 GPU cores)
- Accelerator count: Not reported
- Accelerator memory / system memory: Not reported / Not reported
- Model filename/artifact label: Not reported
- Model / quantization: Llama 7B (source label; version unverified) / mostly Q4_0
- Runtime / backend: llama.cpp / llama-bench 8e672ef (build 1550) / Metal
- Test description: llama-bench separate tg128 test; flash attention not reported
- Prompt tokens / generated tokens: Not reported / 128
- Batch size / context setting / concurrency: Not reported / Not reported / Not reported
Source-specific notes:
- Externally reported by the original tester; not measured or reproduced by HighVRAM Hardware Watch.
- Publication date is the comment date. Exact benchmark execution date is unknown.
- Prompt processing and generation are separate microbenchmark tests; pp512 is not the prompt paired with tg128.
- Context capacity, prior context depth, concurrency and repetition count are not reported. Do not infer them from the test label.
- The exact source row identifies llama 7B. Model version and artifact are unverified; the surrounding discussion protocol does not establish them for this run.
- Reported model size: 3.56 GiB; parameters: 6.74 B. Model size is not peak memory.
- GPU layer setting (ngl): 99.
- Reported dispersion: ±0.08 tokens/s; the comment does not define its statistical interpretation.
- Different runtime commits, flash-attention settings, power limits and undisclosed host settings prevent a universal cross-source hardware ranking.
- Original quantization label: mostly Q4_0.
- Memory capacity, exact model artifact, operating-system version, flash-attention state and power limit are not reported in this post.
- Batch size is not stated in the original 2023 report. The subsequently updated thread methodology cannot establish this run setting; batch_size is null.
H20: Qwen-team Qwen3-32B AWQ-INT4
Source type: First-party model-project results from the QwenLM/Qwen3 team on NVIDIA H20. These are not NVIDIA-authored tests. Read the pinned Qwen-team benchmark table. The file was first published May 16, 2025; the pinned revision is dated May 20, 2025. Execution date: not reported. Accessed October 9, 2026.
Reported configuration: One NVIDIA H20 (source capacity label: 96GB), Qwen3-32B AWQ-INT4, Hugging Face Transformers 4.51.3, PyTorch 2.6.0+cu124, AutoAWQ 0.2.9 and AutoAWQ kernels 0.0.9. Batch size 1; 2,048 output tokens at each input length. Backend, KV-cache type, flash-attention setting, concurrency, configured context capacity and repetitions are not reported for these selected profiles.
Comparison limit: Combined throughput is (prompt tokens + generation tokens) / time. The precise timing boundary is unknown. It is not decode speed: a rising rate with longer input does not establish faster generation. Four source profiles produce eight metric observations; they are not eight independent runs.
Combined input + output throughput (tokens/s)
| Reported tokens | Combined tokens/s | Source & profile |
|---|---|---|
| 1 input 2,048 output | 41.8 | Original Profile |
| 6,144 input 2,048 output | 68.71 | Original Profile |
| 14,336 input 2,048 output | 107.02 | Original Profile |
| 30,720 input 2,048 output | 188.11 | Original Profile |
Reported GPU memory (source MB)
Memory limit: The table labels this column “GPU Memory (MB),” while the introduction says GB. Values below preserve the table’s MB label without conversion. Decimal/binary convention, measurement boundary and peak status are all unknown; these are not confirmed peak-memory measurements.
| Reported tokens | Source MB | Source & profile |
|---|---|---|
| 1 input 2,048 output | 19,109 | Original Profile |
| 6,144 input 2,048 output | 20,795 | Original Profile |
| 14,336 input 2,048 output | 23,105 | Original Profile |
| 30,720 input 2,048 output | 27,718 | Original Profile |
Source locators and profile identifiers
All four profiles share configuration-a468c43a2361bdce. Input/output counts describe workload lengths, not configured context capacity. The exact model artifact, revision and hash are unreported. The download retains full source notes and unknown fields.
- 1 input / 2,048 output: profile-e9f7d994e603a957. AWQ-INT4 source row, line 757. Throughput record; memory record.
- 6,144 input / 2,048 output: profile-b16ec8a577f90bdb. AWQ-INT4 source row, line 766. Throughput record; memory record.
- 14,336 input / 2,048 output: profile-8e3d9c079255c648. AWQ-INT4 source row, line 775. Throughput record; memory record.
- 30,720 input / 2,048 output: profile-54ebb76758598206. AWQ-INT4 source row, line 784. Throughput record; memory record.
Two RTX 3090 GPUs: chibop1’s Llama-3.3-70B report
Source type: Community measurements by original tester chibop1 on r/LocalLLaMA. Read the original report. Published December 14, 2024; edited December 16, 2024. These are source dates, not verified execution dates. Execution date: not reported. Accessed October 9, 2026.
Reported configuration: Two NVIDIA GeForce RTX 3090 GPUs; Llama-3.3-70b-instruct, q4_K_M (source spelling); llama.cpp b4326; flash attention on, temperature 0 and seed 1000. Backend, per-GPU memory capacity, interconnect, host specifications, batch size, concurrency, context capacity and repetitions are unreported. Product and runtime names do not fill those gaps.
Keep the KV regimes separate: The source says q8_0 KV quantization became necessary around 20k context. The selected 20,234- and 32,170-prompt-token rows use q8_0; earlier KV type is unspecified. They have separate configuration IDs. Outputs also vary, so these observations do not establish a controlled context-scaling test or a cross-hardware ranking.
Reported generation throughput (tokens/s)
These five observations use the source’s “Generation Speed” column, confirmed by the author as tokens/second. Exact timing boundaries are unreported. Prompt-speed and total-wall-clock columns are not imported as additional observations.
| Reported tokens & KV | Generation tokens/s | Source & profile |
|---|---|---|
| 258 prompt 576 output KV: not reported | 17.87 | Original Profile |
| 8,013 prompt 847 output KV: not reported | 14.91 | Original Profile |
| 18,209 prompt 1,065 output KV: not reported | 11.84 | Original Profile |
| 20,234 prompt 862 output KV: q8_0 | 10.05 | Original Profile |
| 32,170 prompt 714 output KV: q8_0 | 7.88 | Original Profile |
KV configurations and profile identifiers
- Earlier KV type unspecified: configuration-925473990ae821b9 covers prompt counts 258, 8,013 and 18,209. Unknown is not evidence that KV quantization was disabled.
- q8_0 KV: configuration-3891e8b11c18dede covers prompt counts 20,234 and 32,170. Do not merge across this switch.
- 258 prompt / 576 output: profile-9a4c1be8e4570a42. Source locator: RTX3090 result-table row with Prompt Tokens=258. Generation record.
- 8,013 prompt / 847 output: profile-1ce4b30a8e26d72f. Source locator: RTX3090 result-table row with Prompt Tokens=8013. Generation record.
- 18,209 prompt / 1,065 output: profile-fd7ab0b16f8a611a. Source locator: RTX3090 result-table row with Prompt Tokens=18209. Generation record.
- 20,234 prompt / 862 output: profile-0083e57397aaaf49. Source locator: RTX3090 result-table row with Prompt Tokens=20234. Generation record.
- 32,170 prompt / 714 output: profile-2357452d68b2d98b. Source locator: RTX3090 result-table row with Prompt Tokens=32170. Generation record.
“One-shot” describes the interaction shape, not the repetition count. Prompt lengths do not establish context capacity. The exact model artifact, revision and hash are unreported. The download retains full source notes and unknown fields.
Source index and interpretation notes
Jump by hardware or metric
These links jump to the original source sections, keeping metric definitions and reported settings in context. They do not rank hardware.
Hardware
- Apple M2 Max Mac Studio: brozkrut: M2 Max
- Apple M3 Ultra Mac Studio: Rapid-MLX: M3 Ultra
- Apple M4 Max: Official MLX-LM: M4 Max
- NVIDIA H20 (1 GPU): Qwen team: H20
- RTX 3090 (GPU count unreported): RodriMora: RTX 3090 / 5090
- RTX 3090 (2 GPUs): chibop1: two RTX 3090 GPUs
- RTX 4090 (1 GPU): lhl: RTX 4090
- RTX 5090 (GPU count unreported): RodriMora: RTX 3090 / 5090
Metric
- Prompt processing (reported): Official MLX-LM: M4 Max, lhl: RTX 4090, RodriMora: RTX 3090 / 5090, brozkrut: M2 Max
- Rapid prompt/TTFT proxy: Rapid-MLX: M3 Ultra
- Generation / decode: Rapid-MLX: M3 Ultra, Official MLX-LM: M4 Max, lhl: RTX 4090, RodriMora: RTX 3090 / 5090, brozkrut: M2 Max, chibop1: two RTX 3090 GPUs
- Streamed time to first token: Rapid-MLX: M3 Ultra
- Combined input + output throughput: Qwen team: H20
- Reported GPU memory (source MB): Qwen team: H20
Snapshot 3: 43 externally reported metric observations from seven source publications. The collection contains 13 prompt-throughput observations (including four serving-level proxies), 18 generation/decode observations, four streamed time-to-first-token observations, four combined input + output throughput observations and four reported GPU-memory observations in source-labeled MB. Sources were published from 2023 to 2026; accessed October 8–9, 2026. Publication and edit dates are source metadata, not verified execution dates.
This is a browsable database snapshot with downloadable structured data. Browse controls, when supported, narrow the source tables; the jump links work without scripts. This is a fixed snapshot with no automatic updates. Multiple metric records can describe the same reported request group or configuration; 43 observations do not mean 43 independent benchmark runs.
Browse by original source
- M3 Ultra: Rapid-MLX streaming requests, September 2, 2026 (12 metric observations)
- M4 Max: official MLX-LM benchmarks, October 15, 2025 (4 metric observations)
- RTX 4090: lhl, August 6, 2025 (4 metric observations)
- RTX 3090, RTX 5090: RodriMora, August 2, 2025 (8 metric observations)
- M2 Max Mac Studio: brozkrut, November 23, 2023 (2 metric observations)
- H20: Qwen-team Qwen3-32B AWQ-INT4, May 2025 (8 metric observations)
- Two RTX 3090 GPUs: chibop1, December 2024 (5 metric observations)
- Download this snapshot
Read the results in context
- Source-specific metrics: Prompt processing, generation/decode, streamed time to first token, combined input + output throughput and reported GPU memory use separate tables and units. Read the definition in each source section before comparing any values.
- Rapid-MLX prompt/TTFT proxy: Reported prompt tokens divided by streamed TTFT includes serving overhead. It must not be treated as an isolated engine-prefill rate or compared directly with the MLX-LM and llama-bench prompt-processing rates.
- llama-bench phase labels: In the historical CUDA and Metal sections, pp512 is the 512-token prompt-processing microbenchmark and tg128 is the separate 128-token generation microbenchmark. They do not describe one combined prompt/response run.
- Unknowns stay unknown: A prompt length is not a context-capacity setting. Product names, software defaults and quantization aliases do not fill missing configurations. Generic memory columns and after-sweep allocator peaks do not establish per-profile peak memory.
- Precision and uncertainty: Values retain the source’s reported precision. The ± values in the older llama-bench reports are reproduced as printed; those posts do not define their statistical interpretation.
- Scope: These reports do not establish an overall hardware ranking, answer quality, current performance guarantees or performance per watt. The downloadable records preserve their citations, definitions and limitations.
Download this snapshot
Snapshot 3 contains the 43 observations on this page, with source locators, reported configurations, source-specific metric definitions and explicit unknowns. Null in JSON, SQL NULL and blank numeric fields in CSV mean “not reported or not established,” never zero.
Snapshot identifier: cited-2aca936bd67d79e8. The export/data-contract version is 0.2.0, separate from the snapshot number. New records use the highvram.metric-extension.v1 contract. The 30 legacy JSON records retain their original metric keys; absent new keys mean unknown, not zero.
Download Snapshot 3 (ZIP). Includes only CSV, JSON, SQLite and a README. JSON and SQLite preserve source-specific metric definitions alongside the records.
This update preserves all 30 observations from Snapshot 2 and adds 13 observations from two sources: eight Qwen-team H20 observations and five two-RTX-3090 generation observations. The additions represent nine source profiles, not nine proven independent runs. Download the archived Snapshot 2 (ZIP).
Test the workload you care about
Turn a promising result into a checkable shortlist:
- Save the evidence. Copy the record identifier, source link, exact model and quantization, runtime version and reported test settings into your test record. Keep unreported settings marked unknown; a faster number from a different test is not proof of a better match.
- Set your own pass/fail targets. Test answer quality, time to first response, completion time and memory behavior with the context length and simultaneous requests you actually need. Use the workload test before treating a benchmark as a reason to buy.
- Check the exact hardware, then the offer. Use the Hardware guide to review the options and Deals to reach the matching offer page. A shared product-family name does not establish the same memory, configuration or performance. Confirm system compatibility, delivered cost and seller terms before buying.
Found a transcription error or missing context? Use the Community correction checklist with the record identifier, source link and proposed correction.
Ran a comparable test? Build a benchmark submission. For future additions, follow the weekly update by RSS.
Take it back to the community
Help improve the next result. New to this? Ask with your goal, current hardware and blocker. Experienced readers can share a reproducible build or useful finding, or flag a source or record correction.
Find the weekly workshop, then return to your original thread with the outcome. A helpful contribution earns thanks, not an automatic evidence label.
Want a concise weekly selection of useful experiments, hardware developments and community discussions? See HighVRAM Weekly Digest and its signup status. Email signup is not open yet; RSS remains available.
Planning around the Qwen3-30B-A3B profile?
Use the Qwen3-30B-A3B VRAM guide for official artifact sizes and cache arithmetic, then compare 24GB, 32GB and 48GB hardware paths. The preserved M4 Max profile is one source configuration, not a cross-platform ranking.
Open evidence gap: we do not yet have a controlled same-model RTX 5090 versus DGX Spark result. Use the capacity-first comparison, then submit reproducible measurements rather than inferring speed from memory or peak compute.
Open compact-system evidence gap: we do not yet have a controlled same-model DGX Spark versus Mac Studio result. Use the capacity-first comparison, then submit a reproducible pair.
Open Blackwell evidence gap: we do not yet have a controlled same-workload DGX Spark versus RTX PRO 6000 result. Use the capacity-first comparison, then submit a reproducible pair.
DGX Spark alternatives: Use the alternatives hub to define the missing controlled comparison in the constraint-first comparison hub.
DGX Spark vs H100: Use the capacity-first guide to define a controlled DGX Spark/H100 test in the exact capacity and platform comparison.
DGX Spark vs H200: Use the capacity-first guide to define a controlled DGX Spark/H200 test in the exact 128GB-versus-141GB comparison.
Evidence gap: the database does not yet contain a controlled DGX Spark-versus-A100 pair. Use the comparison protocol and submit matching artifact, runtime, context, concurrency, timing, exact A100 form factor/MIG state and peak-memory fields.
Evidence gap: there is not yet a controlled RTX 4090-versus-DGX Spark pair in this fixed database. Use the comparison protocol and submit matching artifact, runtime, context, concurrency, timing and peak-memory evidence.
Evidence gap: the database does not yet contain a controlled RTX 3090-versus-DGX Spark pair. Use the comparison protocol and submit matching artifact, runtime, context, concurrency, timing, device placement and per-device peak memory.
Choose the pair before collecting evidence: use the local AI GPU comparison directory, then build or submit a matched benchmark record for the same workload.