Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing.

arXiv:2404.16914v2 Announce Type: replace-cross Abstract: MoE facilitates the development of large models by making the computational complexity of the model no longer scale linearly with increasing parameters. The learning sparse gating network selects a set