Launch Overhead and Fusion

The idea in one minute#

Every kernel launch has a fixed cost of a few microseconds — the host prepares arguments, the driver validates them, a command crosses PCIe. For a kernel that runs for milliseconds, that is nothing. For a program made of thousands of tiny kernels, the fixed costs add up to more than the work, and the GPU sits idle between launches.

Two cures: fusion (combine several small operations into one kernel) and graph capture (record a whole sequence of launches once and replay it with a single call).

An analogy#

Sending a parcel has a fixed cost: the label, the queue at the post office. Sending one large box is efficient. Sending a thousand envelopes, each with one sheet of paper, each with its own trip to the post office, is mostly queueing.

Fusion puts the sheets in one envelope. Graph capture gives the post office a standing order.

A picture#

flowchart TB
  subgraph A["Unfused: 3 launches, data read and written 3 times"]
    direction LR
    X1[("x")] --> K1["multiply"] --> T1[("temp")] --> K2["add"] --> T2[("temp")] --> K3["relu"] --> Y1[("y")]
  end
  subgraph B["Fused: 1 launch, data read once, written once"]
    direction LR
    X2[("x")] --> KF["multiply, add, relu<br/>in registers"] --> Y2[("y")]
  end
  class X1,X2,Y1,Y2 memory
  class T1,T2 warn
  class K1,K2,K3,KF compute

How it really works#

Where the time goes#

A launch costs roughly 3–10 µs on the host side with plain CUDA; reaching it through a Python framework can add 10–50 µs of interpreter and dispatch work per operation.

A transformer layer is around 10–20 operations. A model with 32 layers needs several hundred launches per token. At 10 µs each that is 3–5 ms of pure overhead per token — comparable to the entire memory-bound compute time from II.03. For small models the overhead is the runtime, and the GPU shows low utilization while the CPU core driving it is pinned at 100%.

That signature — GPU not busy, one CPU core saturated — is the diagnosis for being launch-bound.

Fusion#

Combining y = relu(a*x + b) into one kernel wins twice:

1. One launch instead of three.
2. The intermediates never go to memory. Unfused, each step reads and writes a full array; fused, each element is read once, transformed in registers, written once. For a memory-bound operation that alone is a 3x reduction in traffic.

Elementwise chains fuse trivially. Fusing into a matrix multiply (for example its bias and activation) is what optimized libraries and compilers (TensorRT, torch.compile, XLA) do for you.

Graph capture#

When the sequence of kernels is the same every step — as in an LLM decode loop — you can record it once. A CUDA graph captures a series of launches with their arguments; replaying it submits the whole series with one call. Per-launch host cost falls from microseconds to a fraction of one.

The catch: the captured sequence is frozen. Shapes and control flow must not change, so servers capture a graph per batch-size bucket and pad to the nearest one.

Batching amortizes it too#

Launch overhead is per launch, not per element. Doubling the batch size doubles the work per launch for the same overhead. This is a second, independent reason batching helps (the first was the roofline).

Code#

// fusion.go — the same arithmetic as three passes and as one.
package main

import (
	"fmt"
	"time"
)

func timeIt(f func()) time.Duration {
	f()
	t0 := time.Now()
	for i := 0; i < 5; i++ {
		f()
	}
	return time.Since(t0) / 5
}

func main() {
	const n = 50_000_000
	x, t1, t2, y := make([]float32, n), make([]float32, n), make([]float32, n), make([]float32, n)
	for i := range x {
		x[i] = float32(i%100) - 50
	}
	const a, b = 1.5, 0.5

	unfused := timeIt(func() {
		for i, v := range x { // kernel 1: multiply
			t1[i] = a * v
		}
		for i, v := range t1 { // kernel 2: add
			t2[i] = v + b
		}
		for i, v := range t2 { // kernel 3: relu
			y[i] = max(v, 0)
		}
	})
	fused := timeIt(func() {
		for i, v := range x { // one kernel: intermediates stay in registers
			y[i] = max(a*v+b, 0)
		}
	})

	fmt.Printf("unfused: %v  (24 bytes moved per element)\n", unfused)
	fmt.Printf("fused:   %v  ( 8 bytes moved per element)\n", fused)
	fmt.Printf("speedup: %.1fx\n\n", float64(unfused)/float64(fused))

	// Launch overhead: one decode step of a small model.
	const launches, perLaunch, compute = 400, 10e-6, 2e-3
	fmt.Printf("launch overhead per token: %.1f ms of %.1f ms total (%.0f%%)\n",
		launches*perLaunch*1e3, (launches*perLaunch+compute)*1e3,
		100*launches*perLaunch/(launches*perLaunch+compute))
}

The CPU shows the memory-traffic half of the story. On a GPU you also save two launches.

Remember this#

• Each launch costs microseconds of host time; hundreds per token add up.
• Launch-bound signature: idle GPU, one saturated CPU core.
• Fusion removes launches and intermediate memory traffic.
• Graph capture replays a fixed sequence of launches with one call.

Try it#

1. Run fusion.go. Is the speedup close to 3x? Explain any difference.
2. In the overhead calculation, how small must perLaunch become for overhead to drop below 10%? Which technique gets you there?
3. A model serves batch sizes from 1 to 64. Graphs need fixed shapes. Design a bucketing scheme and say what it costs.

Check yourself#

1. How do you recognise a launch-bound workload from monitoring data?
2. Give the two separate benefits of fusion.
3. What restriction does graph capture impose?