Copy the Same, Distill the Difference: Initializing Linear Vision Transformers.

arXiv:2609.35745v1 Announce Type: cross Abstract: Linear Vision Transformers (ViTs) are designed to replace the attention in Softmax ViTs with the linear-complexity attention operator for more efficient token routing, but they require from-scratch pre-training