RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing.

arXiv:2610.11775v1 Announce Type: new Abstract: Sparse Mixture of Experts (MoE) models scale more efficiently than dense models by routing tokens to modular expert networks that are only active for processing a fraction of tokens. A leading hypothesis for the