boltbeam.fyi

What is a roofline? The fastest your GPU can run an LLM

Decode speed for one user is capped by bytes per token over memory speed. How to compute the roofline, and why kernels fall short of it: Little's law and Volkov's latency hiding, with RTX 5090 examples.

When a local model writes for one user, each new token reads every weight in the model once. Memory is the bottleneck, not math. That gives a hard ceiling:

fastest possible time per token = bytes read per token ÷ memory speed

That ceiling is the roofline. No engine, no setting, no kernel goes under it. Everything above it is time an engine can still win.

A worked example

Qwen3.8-27B in a 4-bit GGUF, on an RTX 5090:

StepValue
Bytes read per token15.74 GB
Memory speed, measured on this card1,710 GB/s
Floor per token15.74 ÷ 1,710 = 9.20 ms
Roofline108.7 tokens per second
Measured, llama.cpp81.9 tokens per second, 75% of the roofline

The memory speed is measured on the GPU itself, not read off a spec sheet. The 5090's sheet says 1,792 GB/s. The card delivers 1,710.

Two things that move the roofline

Fewer bytes. A smaller quant reads fewer bytes per token, so the floor drops. That is why 4-bit is faster than 16-bit: on an RTX 3090, Qwen3-8B went from 54.0 tokens per second in BF16 to 124.5 in Q4_K_M.

More tokens per read. Speculative decoding drafts several tokens and checks them in one pass, so one read of the weights yields more than one token. On the Qwen3.8-Flash-Next result, a step reads 5.76 GB and yields 3.38 tokens: a 6.4 ms floor, a roofline of 528 tokens per second.

Why kernels fall short: Little's law and Volkov

The roofline assumes the memory is busy every moment. Two results say what that takes.

Little's law. In any system that runs steadily, the number of things inside equals the rate they arrive times how long each one stays: L = λW (Little, 1961). For GPU memory, that reads:

bytes in flight = memory speed × how long each read waits

At 1,710 GB/s, an RTX 5090 must have about 1.7 MB of reads in flight for every microsecond a read waits. Less than that and the memory sits idle, and the kernel runs under the roofline no matter how good its math is.

Volkov. Vasily Volkov's dissertation is about how a GPU kernel gets that much in flight (Volkov, 2016). Enough independent work has to be running to cover the wait, and more threads is only one way to get it: each thread can also keep several reads going at once. He found that the earlier GPU performance models misjudged real throughput by up to 1.7 times, which is why every number on this site is measured rather than modelled.

How they connect to the roofline. The roofline is the ceiling. Little's law is the price of reaching it. Volkov is how a kernel pays it. And every kernel pays a fixed toll on top: at the start, the first reads wait with nothing in flight yet to hide them; at the end, the last ones drain with nothing left to overlap. That toll is a few microseconds, whatever the kernel's size.

So big kernels reach the roofline and small ones cannot. Qwen3.8-27B in 4 bits, llama.cpp, RTX 5090, one call of each kernel:

Table one kernel readsSizeShare of memory speed
Output head1,043 MB99%
Feed-forward matrices50 to 90 MB86 to 94%
Attention key and value4 to 6 MB57 to 67%

The output head's floor is 610 microseconds, and the toll is lost in it. The key and value tables have a floor of 2.5 to 3.3 microseconds, and the toll is a third to nearly half of their time. BoltBeam labels these rows "too small to fill memory".

The same law explains the small kernels between the matrices, which read almost nothing. llama.cpp's input conversion runs as 20 blocks of work on a GPU with 170 processors, 461 times a token: it can never put 1.7 MB in flight, so it pays the toll and little else. That is why, at one user, fewer and fused kernels win. The longer write-up, with the same tie-out on an Apple M3, is in The roofline holds on a second GPU.

Reading the gap

Every result page splits the time above the floor two ways:

  • Per role: each part of the model (attention, feed-forward, output head) against its own floor.
  • Per kernel: each GPU program the engine launched, against the floor of the weights it reads. Kernels that read no weights, like norms and conversions, have no floor: all of their time is gap.

On a consumer card, engines reach 73 to 88% of the roofline. On an H100 at one user, about 40%. The GPU guide explains why.

Sources