Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models.

arXiv:2609.37751v1 Announce Type: new Abstract: Sparse mixture-of-experts (MoE) large language models scale model capacity by routing each token to a small subset of experts. Their routers are regularized with load balancing terms and learn affinity scores thr