boltbeam.fyi

llama.cpp vs vLLM for one user: measured on 10 GPU and model pairs

Same model, same GPU, one user: vLLM with AWQ against llama.cpp with Q4_K_M. vLLM led all 10 pairs, by 1 to 11%. The matrix kernels tie; the gap is the small kernels between them.

Same model, same GPU, one user. llama.cpp with a Q4_K_M GGUF against vLLM with an AWQ checkpoint. Both are 4-bit, and they read almost the same bytes per token.

ModelGPUllama.cpp Q4_K_MvLLM AWQvLLM ahead
Qwen3.8-27BRTX 309041.546.2+11%
Qwen3.8-27BL40S40.043.6+9%
Qwen3.8-27BRTX 409050.454.4+8%
Qwen3-14BRTX 309076.482.4+8%
Qwen3.8-27BRTX 509081.987.5+7%
Qwen3.8-27BH100 80GB HBM383.687.8+5%
Qwen3-14BL40S74.177.7+5%
Qwen3-14BRTX 409094.497.7+3%
Qwen3-14BH100 80GB HBM3144.4147.6+2%
Qwen3-14BRTX 5090149.5151.5+1%

vLLM led every pair, by 1 to 11%. Not 2x. A few percent. All of these are one card; a two-card split changes how the model is laid out, which is a different comparison.

Why vLLM is ahead

You might expect the 4-bit matrix kernels to explain it. They do not. On Qwen3.8-27B on an RTX 5090, llama.cpp's matrix kernels and vLLM's Marlin kernels run at the same share of memory speed: 87 to 90%. They tie.

The gap is the small work between the matrices:

  • llama.cpp converts the input before every 4-bit matrix. Its kernels do their math in 8-bit integers, so a separate quantize_q8_1 kernel runs first. On the 27B that is 461 launches and 0.42 ms per token. vLLM's Marlin takes 16-bit input directly and has no such step.
  • vLLM fuses the small kernels. It merges the residual add and the norm into one kernel. llama.cpp runs them separately.

Count the kernels: llama.cpp launches 1,914 per token, vLLM 790. At one user, each kernel has a fixed cost to start, whatever it does. Fewer kernels, less time. The 27B result page lists every one of them.

Why llama.cpp converts at all

It was the right call when it was written. In 2023 the switch to 8-bit integer math made llama.cpp decode 1.34 to 1.70x faster on an RTX 3090 (llama.cpp PR #2067). The kernels then were far from memory speed, so cheaper math paid. Today they are near it, and the conversion is mostly cost. AMD's llama.cpp fork measured dropping it on one GPU family: +3.09% on average across 38 models (PR #61).

Which should you use?

  • The model fits, one user, 4-bit: vLLM is 1 to 11% faster on every card here.
  • The model does not fit in VRAM: llama.cpp. It can keep part of a model in system RAM; vLLM is built to keep everything on the GPU.
  • Many users at once: vLLM's scheduler is built for it. This site measures one user.