Optimal Transport Aggregation for Distributed Mixture-of-Experts.

arXiv:2312.09877v3 Announce Type: replace-cross Abstract: This paper considers the setting in which local Mixture-of-experts (MoE) models are fitted independently on distributed datasets and must be combined into a single global MoE model. Weighted averaging t