boltbeam.fyi

Qwen3-8B BF16 on RTX 3090

safetensors BF16

Qwen's BF16 safetensors; llama.cpp ran unsloth's BF16 GGUF of the same weights (unsloth/Qwen3-8B-GGUF).

Engines

vLLM

0.31.0 ↗measured 2026-11-09 02:47 UTC

50.2 tok/s

85% of the roofline58.9 tok/s possible

now, a token
19.93 ms
floor
16.99 ms
read, a token
15.16 GB

TensorRT-LLM

1.2.1 ↗measured 2026-11-09 02:41 UTC

51.4 tok/s

87% of the roofline58.9 tok/s possible

now, a token
19.44 ms
floor
16.99 ms
read, a token
15.22 GB

llama.cpp

b11433 · 50569eb87 ↗measured 2026-11-09 02:50 UTC

54.0 tok/s

92% of the roofline58.9 tok/s possible

now, a token
18.50 ms
floor
16.99 ms
read, a token
15.22 GB

Speed at context

128 tokens
vLLM50.5 tok/s of 59.1
TensorRT-LLM52.3 tok/s of 59.1
llama.cpp54.3 tok/s of 59.1
512 tokens
vLLM50.2 tok/s of 58.9
TensorRT-LLM51.4 tok/s of 58.9
llama.cpp54.0 tok/s of 58.9
Measured at short contexts only. The pink tick is the roofline at that context.

Where the time goes

Now, a token
18.50 ms
Roofline floor, a token
16.99 ms
Gap to the floor
1.52 ms
Kernels a token
473

Per role ms a token. floor (bytes ÷ measured memory speed) above the floor

attention0.37 / 0.09 ms 23%
attn_qo BF162.91 / 2.70 ms 93%
attn_kv BF160.83 / 0.68 ms 81%
ffn_gate_up BF168.22 / 8.09 ms 98%
ffn_down BF164.16 / 4.05 ms 97%
lm_head BF161.40 / 1.39 ms 99%
kernel gaps0.60 ms · no floor
norm0.44 ms · no floor
KV cache write0.05 ms · no floor
copy0.00 ms · no floor
elementwise0.00 ms · no floor
Per kernel (8): ms now / ms at the roofline, launches a token
mul_mat_vec_f · BF1617.51 / 16.90 ms ×217
flash_attn_ext_f160.31 ms · no floor ×36
rms_norm_f320.28 ms · no floor ×73
rms_norm_mul_rope_f320.16 ms · no floor ×72
flash_attn_stream_k_fixup_uniform0.06 ms · no floor ×36
k_set_rows0.05 ms · no floor ×36
k_get_rows_float0.00 ms · no floor ×2
k_bin_bcast0.00 ms · no floor ×1

No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.

Run 3090-llama-8b-bf16

Tested on

GPU
RTX 3090 · 24 GB
System RAM
63 GB
CPU threads
8
Memory, measured
896 GB/s
Temp, max
see notes
Power
see notes
SM clock
–
Driver
595.112.07
Measured
2026-11-09 02:47 UTC · context 512 tokens

Qwen3-8B on other GPUs

H100 80GBdata center227.0 tok/s · 33%RTX 3090consumer124.5 tok/s · 66%RTX 5090consumer95.9 tok/s · 85%

Best engine and quant on each GPU, one user. The percentage is that GPU's own roofline.