Compute vs Memory

1. What is it?#

Every computation has two costs: doing the arithmetic and moving the data to where the arithmetic happens. One of them takes longer. Whichever it is, that is your bottleneck, and optimizing the other one is wasted effort.

compute-bound:  the arithmetic units are busy; memory has spare bandwidth
memory-bound:   the arithmetic units are idle, waiting for data

Almost all LLM decode is memory-bound. Almost all prefill is compute-bound. Knowing which regime you are in tells you which optimizations can possibly help.

2. Why does it exist?#

Because of a sixty-year divergence in hardware progress known as the memory wall.

                      1990        2010        2024        growth
Peak compute          ~0.1 GFLOP/s  ~1 TFLOP/s  ~1000 TFLOP/s   ~10,000,000x
Memory bandwidth      ~0.1 GB/s     ~150 GB/s   ~3,350 GB/s     ~33,000x

Compute got roughly 300x faster relative to memory. The consequence: a modern GPU can perform hundreds of arithmetic operations in the time it takes to fetch one number from memory.

H100:  990 TFLOP/s ÷ 3.35 TB/s = 296 FLOP per byte

To keep the arithmetic units busy, you must perform ~296 floating-point
operations on every byte you read. In FP16 (2 bytes/number), that's
~600 operations per number.

Reading a weight and using it once — which is exactly what batch-1 decode does — achieves 2 FLOP per 2 bytes = 1 FLOP/byte. You are running the chip at roughly 0.3% of its arithmetic capability. Not because your code is bad; because of physics and economics.

3. Simple analogy#

A world-class chef with a warehouse across town.

The chef can chop an onion in 2 seconds. The warehouse is a 10-minute drive away, and the delivery truck brings one crate at a time.

• If a recipe needs one onion (memory-bound): 10 minutes of driving, 2 seconds of chopping. The chef’s skill is irrelevant. Hire a faster chef → no improvement.
• If a recipe needs 500 onions from one crate (compute-bound): 10 minutes of driving, 17 minutes of chopping. Now the chef is the bottleneck.

The fixes map exactly:

• Bring bigger crates = higher bandwidth (HBM3e, more memory channels).
• Do more with each delivery = batching, kernel fusion, higher arithmetic intensity.
• Keep a pantry near the stove = caches, shared memory, registers.
• Ask for smaller items = quantization (fewer bytes per number).

Notice that three of those four fixes are things you control in software. Only the first requires buying different hardware.

4. Tiny example#

Two operations on the same tensor. Same data, wildly different behavior.

// regimes.go — the same machine, two operations, two different bottlenecks.
package main

import (
	"fmt"
	"time"
)

func timeIt(fn func()) time.Duration {
	fn() // warm up
	t0 := time.Now()
	const iters = 5
	for i := 0; i < iters; i++ {
		fn()
	}
	return time.Since(t0) / iters
}

func main() {
	// --- Operation 1: elementwise. 1 FLOP per element, 8 bytes moved (read + write)
	const N = 100_000_000
	a, out := make([]float32, N), make([]float32, N)
	t1 := timeIt(func() {
		for i, v := range a {
			out[i] = v * 2
		}
	}).Seconds()
	fmt.Printf("elementwise: %7.1f ms  %5.2f GFLOP/s  %5.1f GB/s\n", t1*1e3, N/t1/1e9, 8*N/t1/1e9)

	// --- Operation 2: matmul. Huge FLOPs, modest bytes
	const n = 512
	A, B, C := make([]float32, n*n), make([]float32, n*n), make([]float32, n*n)
	t2 := timeIt(func() {
		for i := 0; i < n; i++ {
			for k := 0; k < n; k++ {
				aik := A[i*n+k]
				for j := 0; j < n; j++ {
					C[i*n+j] += aik * B[k*n+j]
				}
			}
		}
	}).Seconds()
	fmt.Printf("matmul:      %7.1f ms  %5.2f GFLOP/s  %5.3f GB/s\n", t2*1e3, 2*n*n*n/t2/1e9, 3*4*n*n/t2/1e9)
}

On a laptop CPU, one core, plain Go:

elementwise:    25.8 ms   3.87 GFLOP/s   31.0 GB/s   ← limited by memory traffic
matmul:         50.5 ms   5.31 GFLOP/s  0.062 GB/s   ← limited by arithmetic, memory is idle

The same two operations on an A100 GPU (100M elements; 8192×8192 matmul):

elementwise:  0.42 ms    238 GFLOP/s   1900 GB/s   ← at bandwidth ceiling, 1% of FLOPs
matmul:      44.10 ms     24.9 TFLOP/s    18 GB/s  ← at compute ceiling, 1% of bandwidth

The numbers are 100x apart between the two machines, but the pattern is identical. Two operations on the same hardware. One saturates memory and uses 1% of the arithmetic. The other saturates arithmetic and uses 1% of the bandwidth. Neither is “inefficient” — they are in different regimes, and they need different optimizations.

If you tried to speed up the elementwise op by buying a GPU with more TFLOP/s, you would get zero improvement. If you tried to speed up the matmul with faster memory, likewise zero.

5. Technical explanation#

The model#

For any kernel:

T_compute = FLOPs / peak_FLOP_per_sec
T_memory  = Bytes / peak_bytes_per_sec

T_actual ≈ max(T_compute, T_memory)      [in the ideal, perfectly overlapped case]

The max is because modern hardware overlaps loading and computing. If loads take longer, the arithmetic units idle; if arithmetic takes longer, the memory system idles.

Define arithmetic intensity:

I = FLOPs / Bytes        [FLOP per byte]

and the machine’s ridge point:

I_ridge = peak_FLOP_per_sec / peak_bytes_per_sec

Then:

I < I_ridge  →  memory-bound
I > I_ridge  →  compute-bound

Ridge points for common accelerators (FP16 dense, approximate):

Two things to notice. First, ridge points are high and getting higher — hardware is becoming more compute-rich relative to memory, so memory-boundedness gets worse over time. Second, the L40S’s very high ridge point (weak memory relative to compute) makes it a poor choice for batch-1 decode and a fine choice for large-batch prefill — hardware selection is downstream of this number.

Where LLM operations land#

Operation                          Intensity        Regime
────────────────────────────────────────────────────────────────
Decode, batch 1                    ~1               memory ████████
Decode, batch 8                    ~8               memory ████████
Decode, batch 64                   ~64              memory ██████
Decode, batch 256                  ~250             borderline
Prefill, 2048 tokens               ~2000            compute ████████
LayerNorm / RMSNorm                ~1               memory ████████
Softmax                            ~1               memory ████████
Residual add                       ~0.25            memory ████████
Attention (decode, reading KV)     ~1               memory ████████
Attention (prefill)                ~S/4             compute
Sampling                           ~1               memory
Embedding lookup                   ~0               memory (random access!)

Almost everything is memory-bound. The only reliably compute-bound thing in LLM inference is prefill of long prompts and very large batch GEMMs.

The consequence for optimization#

If you are memory-bound, only three things help:

1. Move fewer bytes. Quantization (INT8 halves it, INT4 quarters it), GQA/MLA (smaller KV), sparsity/MoE (read fewer weights per token).
2. Move bytes faster. Better hardware, or better access patterns (coalescing, avoiding strided reads).
3. Do more per byte moved. Batching, kernel fusion, keeping data in registers/SRAM.

If you are compute-bound, different things help:

1. Do fewer FLOPs. Smaller model, sparsity, early exit, shorter sequences.
2. Use faster units. Tensor cores, lower precision (FP8 doubles throughput on Hopper).
3. Improve utilization. Better tiling, avoid tile quantization, avoid warp divergence.

Applying a compute-bound fix to a memory-bound problem yields zero improvement. This is the most common source of wasted optimization effort in the field, and the reason “measure first” is not a platitude here.

6. Under the hood#

Why can’t the GPU just prefetch everything? It tries. The mechanisms:

Latency hiding through massive multithreading. An H100 SM can hold up to 64 warps (2,048 threads) resident. When one warp stalls on a memory load (~400-800 cycles for HBM), the scheduler instantly switches to another warp that has its data. With enough warps, the arithmetic units stay fed even though every individual load is slow. This is why occupancy matters (Section VI.10).

But latency hiding cannot create bandwidth. If every warp needs new bytes and the total demand exceeds 3.35 TB/s, they all wait. Latency is hideable; bandwidth is not.

The memory hierarchy exists to reduce trips to HBM:

Registers      ~256 KB/SM      ~20 TB/s aggregate     1 cycle
Shared/L1      ~256 KB/SM      ~15 TB/s               ~30 cycles
L2 cache       50 MB (H100)    ~7 TB/s                ~200 cycles
HBM3           80 GB           3.35 TB/s              ~500 cycles
Host DRAM      TBs             ~50 GB/s over PCIe     ~10,000 cycles
Disk/network                   ~1-10 GB/s             ~10^6 cycles

Each level down: ~10x slower, ~100x bigger. The art of a fast kernel is keeping the working set one level higher than the naive implementation does. That is literally all FlashAttention does — it keeps attention tiles in shared memory instead of round-tripping the score matrix through HBM.

7. Performance implications#

Work through the canonical calculation, because you will do it constantly:

Llama-70B, FP16, single H100, batch size 1, decode:

Bytes to read per token = 140 GB (all weights)
                        + KV cache read (~0.3 GB at 4k context)
                        ≈ 140.3 GB

Time = 140.3 GB / 3.35 TB/s = 41.9 ms

Max tokens/sec = 1 / 0.0419 = 23.9 tokens/sec

Now: no software optimization can beat 23.9 tokens/sec here. Not a better kernel, not CUDA graphs, not a faster sampler. The only levers are:

Notice these compose. FP8 + batch 32 + 4-way TP gets you into thousands of tokens/sec. Every row of that table is a section of this curriculum.

8. Production implications#

• Hardware selection follows the regime. For latency-critical decode, buy bandwidth (H200 over H100 — same FLOPs, 43% more bandwidth, ~40% faster decode). For throughput batch jobs, buy FLOPs per dollar.
• Quantization is a bandwidth optimization first, a memory-capacity optimization second. People pitch it as “fits in less memory.” The bigger win is usually the 2-4x decode speedup.
• Your monitoring should show which regime each phase is in. Nsight Compute’s dram__throughput.avg.pct_of_peak_sustained_elapsed and sm__throughput.avg.pct_of_peak_sustained_elapsed answer this directly (Section X.08).
• “Add more GPUs” only helps if you add bandwidth in a way the workload can use. Tensor parallelism does (each GPU reads 1/N of the weights). Data parallelism does not reduce per-request latency at all.

9. Common mistakes#

Optimizing FLOPs when memory-bound. Replacing an operation with a mathematically cheaper one that touches the same memory gains nothing. Common with “we reduced the FLOPs by 30%!” announcements that produce no wall-clock change.

Believing marketing TFLOP/s numbers. Vendor peak numbers assume perfect tensor-core utilization, often with structured sparsity, on ideal shapes. Real kernels achieve 40-80% of peak on GEMM and far less on anything else.

Ignoring the read AND write. An elementwise op reads N bytes and writes N bytes. People count only the read and then wonder why they measured half the expected bandwidth.

Assuming higher occupancy means faster. Occupancy helps hide latency. If you are bandwidth-saturated, more warps just means more warps waiting.

Forgetting the KV cache in the byte count. At long context and large batch, KV reads can exceed weight reads. At 32k context, batch 64, a 70B GQA model reads ~10 GB of KV per step on top of 140 GB of weights.

Treating the whole model as one regime. A single forward pass contains compute-bound GEMMs and memory-bound norms and elementwise ops. Profile per-kernel.

10. Hands-on exercise#

A. Compute your ridge point. Look up your GPU’s peak FP16 (or FP32 if no tensor cores) and its memory bandwidth. Compute the ridge point. Write it in numbers.md.

B. Measure both ceilings. Write a bandwidth benchmark (a large copy or a*2) and a compute benchmark (a large square matmul). What fraction of the spec-sheet peak do you achieve for each? (60-90% is typical; if you get 20%, find out why.)

C. Sweep across the ridge. Write a kernel/op whose arithmetic intensity you can vary — for example y = x; for i in range(K): y = y*a + b on a large array, where K controls FLOPs per byte. Sweep K from 1 to 512, plot achieved GFLOP/s vs intensity. You have just drawn your own roofline. Mark the ridge point. Compare to the theoretical value.

D. Classify a real model. Run any LLM at batch 1 and batch 64. Using Nsight Compute or PyTorch profiler, find the top 5 kernels by time in each case. For each, estimate FLOPs and bytes and classify the regime. Which kernels changed regime?

11. Interview questions#

1. Define arithmetic intensity and the ridge point. Compute the ridge point for an H100.
2. Why is LLM decoding memory-bound? Give the arithmetic.
3. Given a memory-bound kernel, list every way to make it faster, in order of typical impact.
4. Why does H200 outperform H100 on decode despite identical peak FLOPs?
5. A colleague reduced a model’s FLOPs by 40% and saw no speedup. Explain, and tell them what to do instead.
6. Why can massive multithreading hide memory latency but not memory bandwidth limits?
7. At what batch size does decode become compute-bound on an A100? Show your reasoning.

12. Further reading#

• [FUNDAMENTAL] Williams, Waterman, Patterson, “Roofline” (CACM 2009)
• [FUNDAMENTAL] Wulf & McKee, “Hitting the Memory Wall” (1995) — short, prescient
• [FUNDAMENTAL] Drepper, “What Every Programmer Should Know About Memory,” parts 2-3
• [ESTABLISHED] Dao et al., “FlashAttention” (2022) §2 — the clearest applied treatment of IO-awareness in ML
• Next: 08 — FLOPs, bandwidth, arithmetic intensity