Llama-3.3-70B-Instruct Q4_K_M on 2× RTX 3090
GGUF Q4_K+Q6_K
Engines
llama.cpp
b11433 · 50569eb87 ↗measured 2026-11-09 02:06 UTC
19.2 tok/s
90% of the roofline21.3 tok/s possible
- now, a token
- 52.19 ms
- floor
- 47.00 ms
- read, a token
- 42.17 GB
Speed at context
128 tokens
512 tokens
Where the time goes
- Now, a token
- 52.19 ms
- Roofline floor, a token
- 47.00 ms
- Gap to the floor
- 5.18 ms
- Kernels a token
- 561
Launch heavy: 1526 kernel launches per token
Per role ms a token. floor (bytes ÷ measured memory speed) above the floor
Per kernel (3): ms now / ms at the roofline, launches a token
No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.
Run 3090x2-llama-70b-layer
Tested on
- GPU
- 2× RTX 3090 · 24 GB each
- System RAM
- 63 GB
- CPU threads
- 8
- Memory, measured
- 896 GB/s
- Temp, max
- see notes
- Power
- see notes
- SM clock
- –
- Driver
- 595.112.07
- Measured
- 2026-11-09 02:06 UTC · context 512 tokens · measured, in-model (nsys): 8 of 8 weight roles timed
Llama-3.3-70B-Instruct on other GPUs
Llama-3.3-70B-Instruct has only been measured on the 2× RTX 3090 so far.