Function Over Form: Distributional Orthogonalization in Mixture-of-Experts with Replica Expert Mechanism.

arXiv:2609.32398v1 Announce Type: new Abstract: The scaling of LLMs increasingly relies on MoE architectures to decouple active computation from total parameter count. However, the efficacy of MoE is often constrained by expert collapse and representation redu