Which GPU is fastest for local LLMs? Measured tok/s, one user
Decode speed of Qwen3-14B and Qwen3.8-27B on RTX 3090, 4090, 5090, L40S, H100 and H200, each against the GPU's measured roofline. An RTX 5090 decodes the 27B as fast as an H100.
For one person chatting with a local model, decode speed is set by one number more than any other: how fast the GPU moves memory. Every token reads every weight once. So the fastest a GPU can possibly go is its memory speed divided by the model's size. That ceiling is the roofline.
The tables below are measured, one user at a time, batch 1. "Share of roofline" is how close each GPU got to its own ceiling.
Qwen3.8-27B, 4-bit
| GPU | Class | Memory, measured | Best engine | tok/s | Share of roofline |
|---|---|---|---|---|---|
| H200 | data center | 4662 GB/s | vLLM (W4A16-AWQ) | 100.8 | 34% |
| 2× Radeon R9700 | consumer | – | vLLM + MTP k=2 (Quark W4A16) | 92.3 | – |
| H100 80GB | data center | 3237 GB/s | vLLM (W4A16-AWQ) | 87.8 | 41% |
| RTX 5090 | consumer | 1710 GB/s | vLLM (W4A16-AWQ) | 87.5 | 80% |
| H200 NVL | data center | 4513 GB/s | llama.cpp (UD-Q4_K_M) | 78.6 | 27% |
| 2× RTX 3090 | consumer | 896 GB/s | vLLM (W4A16-AWQ) | 68.6 | 76% |
| RTX 4090 | consumer | 962 GB/s | vLLM (W4A16-AWQ) | 54.4 | 88% |
| RTX 3090 | consumer | 898 GB/s | vLLM (W4A16-AWQ) | 46.2 | 80% |
| L40S | data center | 762 GB/s | vLLM (W4A16-AWQ) | 43.6 | 89% |
Qwen3-14B, 4-bit
| GPU | Class | Memory, measured | Best engine | tok/s | Share of roofline |
|---|---|---|---|---|---|
| H200 | data center | 4662 GB/s | vLLM (AWQ) | 166.9 | 30% |
| RTX 5090 | consumer | 1710 GB/s | vLLM (AWQ) | 151.5 | 76% |
| H100 80GB | data center | 3237 GB/s | vLLM (AWQ) | 147.6 | 39% |
| H200 NVL | data center | 4513 GB/s | llama.cpp (Q4_K_M) | 135.0 | 26% |
| RTX 4090 | consumer | 962 GB/s | vLLM (AWQ) | 97.7 | 86% |
| RTX 3090 | consumer | 898 GB/s | vLLM (AWQ) | 82.4 | 78% |
| L40S | data center | 762 GB/s | vLLM (AWQ) | 77.7 | 87% |
What the numbers say
An RTX 5090 decodes the 27B as fast as an H100. 87.5 tokens per second against 87.8. The H100 moves memory almost twice as fast (3,237 GB/s against 1,710), but at one user it reaches only about 40% of its roofline. The H200 is the fastest card here at 100.8, and it uses 34% of its roofline; the H200 NVL, 27%.
Consumer cards run close to their ceiling. The 3090, 4090 and 5090 reach 73 to 88% of their roofline. At one user, the small fixed costs of each step (launching kernels, waiting between them) are a small share of a consumer card's token. On a data-center card the token is so short that those same costs dominate.
So for one user, buy memory speed you can use. Data-center GPUs are built to serve many users at once. That is where their bandwidth pays off. For a single chat, a 5090 gives the same tokens per second as an H100 for this model.
What this does not cover
- One user only. With many users at once, the data-center cards pull ahead.
- Models that fit in VRAM. A model larger than the card is a different problem: see the Qwen3.8-Flash-Next result, which runs a 58.4 GB model on a 32 GB card.
- One run per point. Read a few percent of slack into every number.