RTX 5090 for local LLMs
8 measured runs. 32 GB of VRAM; memory measured at 1710 GB/s, which sets the roofline for every model below.
| Model | Quant | Engine | tok/s | Roofline | Share |
|---|---|---|---|---|---|
| Nemotron-3.5-Lightning-30B | NVFP4 | vLLM | 378.9 | 686.3 | 55% |
| Qwen3-14B | AWQ | vLLM | 151.5 | 200.8 | 76% |
| Qwen3-14B | Q4_K_M | llama.cpp | 149.5 | 197.8 | 76% |
| Qwen3-8B | BF16 | TensorRT-LLM | 95.9 | 112.4 | 85% |
| Qwen3.8-27B | W4A16-AWQ | vLLM | 87.5 | 109.7 | 80% |
| Qwen3.8-27B | UD-Q4_K_M | llama.cpp | 81.9 | 108.7 | 75% |
| Qwen3.8-Flash-Next | GSQ-RCO IQ1_M | llama.cpp + mods | 226.1 | 528.5 | 43% |
| Qwen3.8-Flash-Next | GSQ-RCO IQ1_M | llama.cpp | 77.2 | 537.1 | 14% |
Share = measured tok/s over the roofline: the bytes a token reads over this GPU's measured memory speed. Open a quant for the test machine and the per-role and per-kernel breakdown.