boltbeam.fyi

Llama-3.3-70B-Instruct Q4_K_M on 2× RTX 3090

GGUF Q4_K+Q6_K

Engines

llama.cpp

b11433 · 50569eb87 ↗measured 2026-11-09 02:06 UTC

19.2 tok/s

90% of the roofline21.3 tok/s possible

now, a token
52.19 ms
floor
47.00 ms
read, a token
42.17 GB

Speed at context

128 tokens
llama.cpp19.3 tok/s of 21.3
512 tokens
llama.cpp19.2 tok/s of 21.3
The pink tick is the roofline at that context.

Where the time goes

Now, a token
52.19 ms
Roofline floor, a token
47.00 ms
Gap to the floor
5.18 ms
Kernels a token
561

Launch heavy: 1526 kernel launches per token

Per role ms a token. floor (bytes ÷ measured memory speed) above the floor

ffn_down Q6_K9.51 / 8.60 ms 90%
attention0.87 / 0.19 ms 22%
attn_qo Q4_K7.27 / 6.74 ms 93%
ffn_gate_up Q4_K24.02 / 23.58 ms 98%
ffn_down Q4_K6.22 / 5.89 ms 95%
attn_kv Q4_K0.58 / 0.42 ms 73%
attn_kv Q5_K0.36 / 0.26 ms 72%
attn_kv Q6_K0.40 / 0.31 ms 76%
lm_head Q6_K0.97 / 0.96 ms 99%
norm1.16 ms · no floor
kernel gaps1.04 ms · no floor
quantize0.69 ms · no floor
KV cache write0.11 ms · no floor
copy0.00 ms · no floor
2 other small kernels0.00 ms · no floor
Per kernel (3): ms now / ms at the roofline, launches a token
mul_mat_vec_q · Q4_K40.12 / 36.64 ms ×440
mul_mat_vec_q · Q6_K10.77 / 9.86 ms ×81
mul_mat_vec_q · Q5_K0.57 / 0.26 ms ×40

No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.

Run 3090x2-llama-70b-layer

Tested on

GPU
2× RTX 3090 · 24 GB each
System RAM
63 GB
CPU threads
8
Memory, measured
896 GB/s
Temp, max
see notes
Power
see notes
SM clock
–
Driver
595.112.07
Measured
2026-11-09 02:06 UTC · context 512 tokens · measured, in-model (nsys): 8 of 8 weight roles timed

Llama-3.3-70B-Instruct on other GPUs

Llama-3.3-70B-Instruct has only been measured on the 2× RTX 3090 so far.