Qwen3.8-27B W4A16-AWQ on RTX 5090
safetensors INT4_G128_ZP+BF16
Engines
vLLM
0.31.0 ↗measured 2026-11-09 20:19 UTCnewer release at the time: 0.33.0
87.5 tok/s
80% of the roofline109.7 tok/s possible
- now, a token
- 11.43 ms
- floor
- 9.11 ms
- read, a token
- 15.59 GB
More
- memory, measured
- 1710 GB/s
What was run
python vllm_decode_drive.py --model Qwen3.8-27B-W4A16-AWQ --contexts 512 --batches 1 --tokens 64 --reps 1 --once 64 --max-num-batched-tokens 8192 --model-context-limit 584 --max-num-seqs 16 --capture-pass pairedSpeed at context
128 tokens
512 tokens
Where the time goes
- Now, a token
- 11.43 ms
- Roofline floor, a token
- 9.11 ms
- Gap to the floor
- 2.31 ms
- Kernels a token
- 790
Per role ms a token. floor (bytes ÷ measured memory speed) above the floor
Per kernel (33): ms now / ms at the roofline, launches a token
No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.
Run rtx5090-27b-vllm-batch-b
Tested on
- GPU
- RTX 5090 · 32 GB
- System RAM
- 31 GB
- CPU threads
- 16
- Memory, measured
- 1710 GB/s
- Temp, max
- 77 °C
- Power
- 178 W mean of 600 W
- SM clock
- 2211 MHz, throttled
- Driver
- 595.99.02
- Measured
- 2026-11-09 20:19 UTC · context 512 tokens · measured, in-model (nsys): split by role from the engine's own labels, 9 weight roles
Qwen3.8-27B on other GPUs
H200data center100.8 tok/s · 34%2× Radeon R9700consumer92.3 tok/s H100 80GBdata center87.8 tok/s · 41%RTX 5090consumer87.5 tok/s · 80%H200 NVLdata center78.6 tok/s · 27%2× RTX 3090consumer68.6 tok/s · 76%RTX 4090consumer54.4 tok/s · 88%RTX 3090consumer46.2 tok/s · 80%L40Sdata center43.6 tok/s · 89%
Best engine and quant on each GPU, one user. The percentage is that GPU's own roofline.