Qwen3-14B Q4_K_M on H100 80GB
GGUF Q4_K+Q6_K
Engines
llama.cpp
b11571 · 097cf33d9 ↗measured 2026-11-09 18:19 UTCnewer release at the time: 0.9.0
144.4 tok/s
39% of the roofline374.3 tok/s possible
- now, a token
- 6.93 ms
- floor
- 2.67 ms
- read, a token
- 8.65 GB
More
- memory, measured
- 3237 GB/s
What was run
llama-bench -m Qwen3-14B-Q4_K_M.gguf -p 0 -n 33 -d 512 -ngl 99 -r 1 --no-warmup -o json -ub 512 -b 2048 -ctk f16 -ctv f16 -fa -1Speed at context
128 tokens
512 tokens
Where the time goes
- Now, a token
- 6.93 ms
- Roofline floor, a token
- 2.67 ms
- Gap to the floor
- 4.25 ms
- Kernels a token
- 281
Per role ms a token. floor (bytes ÷ measured memory speed) above the floor
This run timed the whole step only, so there is no per-role split.
Per kernel (2): ms now / ms at the roofline, launches a token
No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.
Run h100-qwen3-14b-q4km-llama
Tested on
- GPU
- H100 80GB · 80 GB
- System RAM
- 443 GB
- CPU threads
- 52
- Memory, measured
- 3237 GB/s
- Temp, max
- 38 °C
- Power
- 143 W mean of 700 W
- SM clock
- 1966 MHz
- Driver
- 580.126.20
- Measured
- 2026-11-09 18:19 UTC · context 512 tokens · measured, in-model (nsys): 7 of 7 weight roles timed
Qwen3-14B on other GPUs
RTX 5090vLLM · AWQ · 4-bit head179.4 tok/s · 76%H200vLLM · AWQ166.9 tok/s · 30%H100 80GBvLLM · AWQ147.6 tok/s · 39%H200 NVLllama.cpp · Q4_K_M135.0 tok/s · 26%RTX 4090vLLM · AWQ97.7 tok/s · 86%RTX 3090vLLM · AWQ82.4 tok/s · 78%L40SvLLM · AWQ77.7 tok/s · 87%
The fastest setup measured on each GPU, one user. The percentage is that GPU's own roofline.