Vision Transformers
Vision Transformers adapt the transformer idea to images by splitting an image into patches and treating those patches like tokens. Attention then models relationships across the whole image.
Images as tokens
- Split the image into fixed-size patches.
- Flatten each patch and project it into an embedding.
- Add positional information so the model knows where patches came from.
- Process the patch sequence with transformer layers.
- Use pooled or special-token representations for classification, retrieval, or downstream tasks.
Why ViTs matter
| Strength | Explanation |
|---|---|
| Global context | attention can connect distant regions directly |
| Scaling | works well with large datasets and model sizes |
| Transfer | pretrained visual encoders adapt to many tasks |
| Multimodal fit | patch tokens can be fused with language tokens |
CNNs still provide strong inductive biases for locality. ViTs trade some of that bias for scale and flexible token-based architecture.
Common variants
- Pure ViT encoders for classification and representation learning.
- Hybrid CNN-transformer models.
- Vision encoders paired with language models.
- Diffusion transformers that replace U-Nets in generative pipelines.
Pitfall
Patch tokens are not words. Small objects, precise geometry, and dense spatial tasks can be hard if patching loses detail or if training data lacks the right supervision.
Connects to: attention · CNNs · transformers