Qwen3.8-Flash-Next GSQ-RCO IQ1_M on RTX 5090
GGUF GSQ-RCO IQ1_M, 2 shards, 58.4 GB
Engines
llama.cpp
v1.0.0 · b1b770d9 ↗measured 2026-11-16 to 2026-11-17
77.2 tok/s
14% of the roofline537.1 tok/s possible
- now, a step
- 44.53 ms
- floor
- 6.40 ms
- read, a step
- 5.76 GB
More
- memory, measured
- 1707 GB/s
- tokens a step
- 3.44 (MTP draft)
- prefill per turn
- 2.61 s
- time to first token
- 2.71 s
- whole session
- 485 s
stock: -c 131072, q8_0 KV, MTP draft head, every other setting defaultllama.cpp + mods
v1.0.0 + 13 mods · b1b770d9 ↗measured 2026-11-16 to 2026-11-17
226.1 tok/s
43% of the roofline528.5 tok/s possible
- now, a step
- 14.96 ms
- floor
- 6.40 ms
- read, a step
- 5.76 GB
More
- memory, measured
- 1707 GB/s
- tokens a step
- 3.38 (MTP draft)
- prefill per turn
- 0.83 s
- time to first token
- 0.93 s
- whole session
- 166 s
-c 131072 -ctk/-ctv q8_0 -fa on -ngl 99 -ncmoe 16 -ub 1024 -b 2048 --moe-cache-mib 6240 -lm none, MTP draft 3 tokens/step, 13 mods on. Settings derived by boltbeam tune for this machine.Speed at context
<32k
32-64k
64-96k
96-128k
Where the time goes
- Now, a step
- 14.96 ms
- Roofline floor, a step
- 6.40 ms
- Gap to the floor
- 8.56 ms
- Kernels a step
- 3648
One third of the gap to the floor is not kernels: ~1.8 ms a step of host idle, plus expert-cache misses.
Per role ms a step. floor (bytes ÷ measured memory speed) above the floor
No kernel capture for this run.
Run flash-next-128k-mods
Tested on
- GPU
- RTX 5090 · 32 GB
- System RAM
- 31 GB
- CPU threads
- 16
- Memory, measured
- 1707 GB/s
- Temp, max
- see notes
- Power
- see notes
- SM clock
- –
- Driver
- 595.99.02
- Measured
- 2026-11-16 to 2026-11-17 · context 131072 tokens · 59-turn session to 129.3k tokens, 3 paired runs, temperature 0
- GPU power capped at 450 W, reset before every run
- start gate: GPU at or under 70 °C, board under 58 °C
- CPU: Ryzen 7 5800X · RAM: 32 GB DDR4-2667 · PCIe 4.0 x16 (B550)
- MoE model larger than VRAM: 16 layers of experts in system RAM
- roofline: 5.76 GB a decode step (3.38 tokens) / 1,707 GB/s measured = 6.4 ms
Qwen3.8-Flash-Next on other GPUs
RTX 5090consumer226.1 tok/s · 43%H200data center100.4 tok/s · 14%2× Radeon R9700consumer72.5 tok/s
Best engine and quant on each GPU, one user. The percentage is that GPU's own roofline.