Qwen3-14B AWQ on H200
awq INT4_G128_ZP+BF16
Engines
vLLM
0.31.0 ↗measured 2026-11-09 22:57 UTC
166.9 tok/s
30% of the roofline547.3 tok/s possible
- now, a token
- 5.99 ms
- floor
- 1.83 ms
- read, a token
- 8.52 GB
Speed at context
128 tokens
512 tokens
Where the time goes
- Now, a token
- 5.99 ms
- Roofline floor, a token
- 1.83 ms
- Gap to the floor
- 4.16 ms
- Kernels a token
- 462
Per role ms a token. floor (bytes ÷ measured memory speed) above the floor
Per kernel (26): ms now / ms at the roofline, launches a token
No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.
Run h200-qwen3-14b-awq-vllm-k8
Tested on
- GPU
- H200 · 140 GB
- System RAM
- 1511 GB
- CPU threads
- 256
- Memory, measured
- 4662 GB/s
- Temp, max
- see notes
- Power
- see notes
- SM clock
- –
- Driver
- 595.91.07
- Measured
- 2026-11-09 22:57 UTC · context 512 tokens · measured, in-model (nsys): split by role from the engine's own labels, 5 weight roles
Qwen3-14B on other GPUs
H200data center166.9 tok/s · 30%RTX 5090consumer151.5 tok/s · 76%H100 80GBdata center147.6 tok/s · 39%H200 NVLdata center135.0 tok/s · 26%RTX 4090consumer97.7 tok/s · 86%RTX 3090consumer82.4 tok/s · 78%L40Sdata center77.7 tok/s · 87%
Best engine and quant on each GPU, one user. The percentage is that GPU's own roofline.