Qwen3.8-27B W4A16-AWQ on 2× RTX 3090
safetensors INT4_G32_ZP+BF16
Engines
vLLM
0.31.0 ↗measured 2026-11-09 02:15 UTC
68.6 tok/s
76% of the roofline90.6 tok/s possible
- now, a token
- 14.58 ms
- floor
- 11.04 ms
- read, a token
- 9.87 GB
Speed at context
128 tokens
512 tokens
Where the time goes
- Now, a token
- 14.58 ms
- Roofline floor, a token
- 11.04 ms
- Gap to the floor
- 3.54 ms
- Kernels a token
- –
Per role ms a token. floor (bytes ÷ measured memory speed) above the floor
This run timed the whole step only, so there is no per-role split.
No kernel capture for this run.
Run 3090x2-vllm-27b-awq-row
Tested on
- GPU
- 2× RTX 3090 · 24 GB each
- System RAM
- 63 GB
- CPU threads
- 8
- Memory, measured
- 896 GB/s
- Temp, max
- see notes
- Power
- see notes
- SM clock
- –
- Driver
- 595.112.07
- Measured
- 2026-11-09 02:15 UTC · context 512 tokens · measured, in-model (nsys): the whole step only, no per-role time
Qwen3.8-27B on other GPUs
RTX 5090vLLM + MTP, 4-bit draft layer · W4A16-AWQ · 4-bit head202.1 tok/s H200vLLM · W4A16-AWQ100.8 tok/s · 34%2× Radeon R9700vLLM + MTP k=2 · Quark W4A1692.3 tok/s H100 80GBvLLM · W4A16-AWQ87.8 tok/s · 41%H200 NVLllama.cpp · UD-Q4_K_M78.6 tok/s · 27%2× RTX 3090vLLM · W4A16-AWQ68.6 tok/s · 76%RTX 4090vLLM · W4A16-AWQ54.4 tok/s · 88%RTX 3090vLLM · W4A16-AWQ46.2 tok/s · 80%L40SvLLM · W4A16-AWQ43.6 tok/s · 89%
The fastest setup measured on each GPU, one user. The percentage is that GPU's own roofline.