Asymptotic Behavior of Transformers.

arXiv:2610.19358v1 Announce Type: cross Abstract: In this paper we study the behavior of transformer networks generating text over long periods of time. We show that even a three-token, single-layer attention-only transformer can fail to converge. We then stud