HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget.

arXiv:2610.34582v1 Announce Type: new Abstract: We describe HobbyLM, a family of small sparse Mixture-of-Experts (MoE) models trained from scratch with a few hundred to a few thousand dollars of rented GPU time. The core is a decoder-only transformer with fine