Qwen3-8B BF16 on RTX 3090
safetensors BF16
Qwen's BF16 safetensors; llama.cpp ran unsloth's BF16 GGUF of the same weights (unsloth/Qwen3-8B-GGUF).
Engines
vLLM
0.31.0 ↗measured 2026-11-09 02:47 UTC
50.2 tok/s
85% of the roofline58.9 tok/s possible
- now, a token
- 19.93 ms
- floor
- 16.99 ms
- read, a token
- 15.16 GB
TensorRT-LLM
1.2.1 ↗measured 2026-11-09 02:41 UTC
51.4 tok/s
87% of the roofline58.9 tok/s possible
- now, a token
- 19.44 ms
- floor
- 16.99 ms
- read, a token
- 15.22 GB
llama.cpp
b11433 · 50569eb87 ↗measured 2026-11-09 02:50 UTC
54.0 tok/s
92% of the roofline58.9 tok/s possible
- now, a token
- 18.50 ms
- floor
- 16.99 ms
- read, a token
- 15.22 GB
Speed at context
128 tokens
512 tokens
Where the time goes
- Now, a token
- 18.50 ms
- Roofline floor, a token
- 16.99 ms
- Gap to the floor
- 1.52 ms
- Kernels a token
- 473
Per role ms a token. floor (bytes ÷ measured memory speed) above the floor
Per kernel (8): ms now / ms at the roofline, launches a token
No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.
Run 3090-llama-8b-bf16
Tested on
- GPU
- RTX 3090 · 24 GB
- System RAM
- 63 GB
- CPU threads
- 8
- Memory, measured
- 896 GB/s
- Temp, max
- see notes
- Power
- see notes
- SM clock
- –
- Driver
- 595.112.07
- Measured
- 2026-11-09 02:47 UTC · context 512 tokens
Qwen3-8B on other GPUs
H100 80GBdata center227.0 tok/s · 33%RTX 3090consumer124.5 tok/s · 66%RTX 5090consumer95.9 tok/s · 85%
Best engine and quant on each GPU, one user. The percentage is that GPU's own roofline.