Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks.

arXiv:2410.11687v4 Announce Type: replace-cross Abstract: In-context learning lets a sequence model adapt to a new task from examples in its input. A prominent line of work shows how self-attention can be constructed to implement gradient descent on a linear p