Vision transformer

A vision transformer (ViT) is a <a href="/facts/Transformer_(machine_learning_model)/cDbjx6a8">transformer</a> designed for <a href="/facts/Computer_vision/Tl2Yyk66">computer vision</a>. A ViT decomposes an input image into a series of patches (rather than text into <a href="/facts/Byte_pair_encoding/lPvX89xK">tokens</a>), serializes each patch into a vector, and maps it to a smaller dimension with a single <a href="/facts/Matrix_multiplication/lYDB6Ro2">matrix multiplication</a>. These vector <a href="/facts/Latent_space/IDYfRCU7">embeddings</a> are then processed by a <a href="/facts/BERT_(language_model)/rDSueM4E">transformer encoder</a> as if they were token embeddings.
ViTs were designed as alternatives to <a href="/facts/Convolutional_neural_network/kHEdnmGU">convolutional neural networks</a> (CNNs) in computer vision applications. They have different inductive biases, training stability, and data efficiency. Compared to CNNs, ViTs are less data efficient, but have higher capacity. Some of the largest modern computer vision models are ViTs, such as one with 22B parameters.
Subsequent to its publication, many variants were proposed, with hybrid architectures with both features of ViTs and CNNs. ViTs have found application in <a href="/facts/Image_recognition/Tl2Yyk66">image recognition</a>, <a href="/facts/Image_segmentation/jofAhbxa">image segmentation</a>, <a href="/facts/Weather_forecasting/nzqyMFCa">weather prediction</a>, and <a href="/facts/Autonomous_driving/KXeFywXS">autonomous driving</a>.

Vision transformer open-in-new

Vision transformer