Qwen3-8B speed on every GPU
6 measured runs across 3 GPUs, fastest first.
| GPU | Quant | Engine | tok/s | Roofline | Share |
|---|---|---|---|---|---|
| H100 80GB | Q4_K_M | llama.cpp | 227.0 | 684.4 | 33% |
| RTX 3090 | Q4_K_M | llama.cpp | 124.5 | 188.9 | 66% |
| RTX 5090 | BF16 | TensorRT-LLM | 95.9 | 112.4 | 85% |
| RTX 3090 | BF16 | llama.cpp | 54.0 | 58.9 | 92% |
| RTX 3090 | BF16 | TensorRT-LLM | 51.4 | 58.9 | 87% |
| RTX 3090 | BF16 | vLLM | 50.2 | 58.9 | 85% |