FlashAttention and efficient attention
Attention is mathematically simple but expensive to run naively. FlashAttention improves attention performance by organizing computation to reduce memory movement between GPU high-bandwidth memory and faster on-chip memory.
The bottleneck
Transformer attention computes interactions between tokens. Naive implementations can materialize large intermediate matrices, which costs memory and bandwidth. On modern GPUs, moving bytes is often the bottleneck, not the arithmetic itself.
What FlashAttention changes
| Idea | Effect |
|---|---|
| Tiling | process attention in blocks that fit on-chip |
| IO awareness | minimize high-bandwidth-memory reads and writes |
| Fused operations | avoid materializing unnecessary intermediates |
| Exact attention | preserve the same mathematical result, not an approximation |
Where efficient attention matters
- Long-context prefill.
- High-throughput serving.
- Training and fine-tuning large transformers.
- Multimodal models with many visual or audio tokens.
- Serving stacks that rely on optimized kernels.
Pitfall
Efficient kernels are workload- and hardware-dependent. Benchmark the actual sequence lengths, batch sizes, model architecture, and serving engine you plan to run.
Connects to: attention · attention maps · hardware basics