1. What is it?#
Splitting the sequence dimension across GPUs, rather than the weight or layer dimension.
TENSOR PARALLELISM: split the weights. Every GPU sees every token.
SEQUENCE PARALLELISM: split the tokens. Every GPU has all the weights
(or a TP slice) but only some tokens. GPU 0 GPU 1 GPU 2 GPU 3
tokens 0-8k tokens 8k-16k tokens 16k-24k tokens 24k-32k
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ KV for │ │ KV for │ │ KV for │ │ KV for │
│ its own │ │ its own │ │ its own │ │ its own │
│ tokens │ │ tokens │ │ tokens │ │ tokens │
└──────────┘ └──────────┘ └──────────┘ └──────────┘2. Problem → Why → Optimization#
PROBLEM At 1M-token context, ONE sequence's KV cache exceeds a single GPU's
memory, and its prefill's quadratic attention exceeds what one GPU
can compute in an acceptable time.
WHY KV scales with sequence length; attention compute scales with S².
TP splits by heads, which caps at n_kv_heads.
OPTIMIZE Split the SEQUENCE across GPUs. Each GPU holds the KV for its
chunk of tokens.
TRADE-OFFS
✓ enables context lengths beyond one GPU's memory
✓ parallelizes the quadratic attention compute
✗ attention needs every query to see every key → cross-GPU communication
in the attention itself
✗ complex; the attention kernel becomes distributed
✗ only worthwhile at very long context
WHEN TO USE Context lengths where a single sequence's KV doesn't fit
(roughly > 200k tokens), or where prefill latency is
unacceptable on one GPU.
WHEN NOT TO Normal context lengths. TP and DP are simpler and sufficient.This is a specialized technique. Most deployments never need it. It appears here because long-context serving is growing and because the mechanism (ring attention) is elegant.
3. Simple analogy#
Four people reading different chapters of a book, who must each answer questions requiring the whole book.
Each person holds one chapter (their KV). To answer a question about the whole book, either:
- pass the question around, each person contributing what their chapter says (ring attention), or
- everyone shares their notes with everyone (all-gather, expensive).
Ring attention is the efficient version: the question circulates, accumulating partial answers, and one full circuit gives the complete answer.
4. Ring Attention — the core algorithm#
Each GPU i holds:
Q_i (queries for its tokens)
K_i, V_i (keys/values for its tokens)
The algorithm:
out_i, m_i, l_i = initialize() # online softmax state
for step in range(N): # N = number of GPUs
compute partial attention: Q_i against the K,V currently held
update out_i, m_i, l_i with online softmax rescaling
SEND K,V to GPU (i+1) mod N
RECV K,V from GPU (i-1) mod N
# after N steps, every Q has seen every K,V
out_i = out_i / l_iThe online softmax (Section III.07) is what makes this work. Because softmax can be computed incrementally with rescaling, each GPU accumulates its partial result as different K,V blocks circulate past, and the final answer is exact.
Communication per layer: each GPU sends its K,V once per ring step
volume = N × (S/N) × n_kv_heads × head_dim × 2 × bytes
= S × n_kv_heads × head_dim × 2 × bytes (independent of N!)
And critically: the SEND can be OVERLAPPED with the COMPUTE of the
current block. With enough compute per block, communication is hidden entirely.That overlap property is why ring attention scales well: the communication is a constant per-layer cost that hides behind the quadratic compute.
5. The variants#
RING ATTENTION
As above. Communication overlaps with compute. Scales to many GPUs.
Best for very long context prefill.
ULYSSES / DeepSpeed sequence parallelism
All-to-All to switch from sequence-split to head-split before attention,
then back after. Attention itself is local (each GPU has all tokens for
some heads).
✓ simpler attention (standard kernels work)
✗ two All-to-Alls per layer
✗ limited by n_kv_heads (can't split beyond it)
MEGATRON SEQUENCE PARALLELISM
Splits the sequence for the NON-attention parts (norms, dropout, residuals),
which are elementwise and need no communication, while keeping TP for the
rest. Reduces activation memory. Used mainly in training.
STRIPED / ZIGZAG ATTENTION
A variant of ring attention that assigns tokens to GPUs in an interleaved
pattern so that CAUSAL masking doesn't leave some GPUs idle.
(In naive ring attention with contiguous chunks, GPU 0's queries only need
GPU 0's keys, while GPU N-1's queries need everyone's — massive imbalance.)The causal-imbalance problem is worth understanding: with contiguous assignment and a causal mask, the last GPU does N times the work of the first. Striped/zigzag assignment fixes it by interleaving, so every GPU has a mix of early and late tokens.
6. Where it fits with other parallelism#
A very-long-context deployment might use:
TP=8 within a node (splits weights and heads)
× SP=4 across nodes (splits the sequence)
= 32 GPUs for ONE sequence
Each GPU holds: 1/8 of the weights, and KV for 1/4 of the tokens for its
1/8 of the heads.Note that this is 32 GPUs serving one user. Sequence parallelism is for the regime where a single request is so large that it needs a cluster. That’s a real regime (1M-token document analysis) but a narrow one.
7. Performance#
1M-token prefill, Llama-3-8B-class model with extended context:
Configuration Prefill time Feasible?
1 GPU — No: KV alone is 131 GB
TP=8 ~180 s Yes but slow; KV is 16 GB/GPU
TP=8 × SP=4 (ring) ~48 s Yes; KV is 4 GB/GPU
TP=8 × SP=8 ~26 s Yes
Ring attention scaling efficiency: 75-85% (communication overlaps well)For decode at long context, sequence parallelism helps less: decode has one query, and the attention over the distributed KV requires a gather of partial results — closer to a FlashDecoding-style split-K across GPUs.
8. Production implications#
- You probably don’t need this. Most deployments cap context at 128k or below, where TP suffices.
- If you do serve 500k+ context, ring attention (or a variant) is the mechanism, and you should expect to dedicate many GPUs to individual requests.
- Use striped/zigzag assignment for causal models. Contiguous assignment wastes most of your GPUs.
- Price long-context requests accordingly. A request needing 32 GPUs for 30 seconds costs what several thousand normal requests cost.
- Consider alternatives first: retrieval (RAG), context compression, sliding window, or simply refusing to support the context length. All are cheaper.
9. Common mistakes#
Using SP when TP would suffice. Unnecessary complexity.
Contiguous token assignment with causal masking. N× load imbalance.
Expecting SP to help decode as much as prefill. Decode has one query; the parallelism opportunity is different.
Not overlapping communication with computation. Ring attention’s efficiency depends on it.
Underestimating the cost. Serving 1M context means dedicating a cluster to one user.
10. Hands-on exercise#
A. Implement ring attention. In PyTorch with torch.distributed, implement ring attention
for 2-4 processes. Verify it matches single-GPU attention exactly. (This exercises everything
from Section III.07.)
B. The causal imbalance. Instrument your implementation to measure per-rank compute time with contiguous assignment. Confirm the imbalance. Implement zigzag assignment and re-measure.
C. Overlap. Measure ring attention with and without overlapping the send/recv with compute. Quantify the benefit.
D. Scaling. Measure prefill time for a long sequence at SP=1, 2, 4. Compute the scaling efficiency.
E. The alternative. For a 500k-token document QA task, compare the cost of (i) full-context inference with SP, and (ii) RAG with 4k of retrieved context. Include quality in the comparison.
11. Interview questions#
- What is sequence parallelism and how does it differ from tensor parallelism?
- Explain ring attention. Why does online softmax make it possible?
- Why is contiguous token assignment bad for causal models?
- How does ring attention’s communication cost scale with the number of GPUs?
- Compare ring attention and Ulysses-style sequence parallelism.
- When would you need sequence parallelism, and what should you try first?
- How would you price a request that requires 32 GPUs for 30 seconds?
12. Further reading#
- [EMERGING] Liu et al., “Ring Attention with Blockwise Transformers for Near-Infinite Context” (2023)
- [EMERGING] Jacobs et al., “DeepSpeed Ulysses” (2023)
- [ESTABLISHED] Korthikanti et al., “Reducing Activation Recomputation in Large Transformer Models” (2022) — Megatron sequence parallelism
- [EMERGING] “Striped Attention” (Brandon et al., 2023) — the causal imbalance fix
- Next: 07 — Collectives and NCCL